Data grading and classifying method and device

Abstract

The embodiment of the invention provides a data grading and classifying method and device, relates to the field of data processing, can realize high-quality data processing, has deep semantic understanding capability, and can construct a dynamic evolution rule system. By introducing technologies such as deep learning and a graph neural network, the intelligent level of the system is improved, and more accurate data classification and more effective knowledge management are realized.

Description

Technical Field

[0001] The present application relates to the field of data processing, and in particular to a method and device for data grading and classification. Background Art

[0002] Data classification is a key link in data governance. Traditional classification methods often rely on manual experience and fixed rules, which are difficult to cope with the processing needs of massive heterogeneous data. The existing data classification system has obvious deficiencies in data quality assessment, semantic understanding and rule management.

[0003] In terms of data quality management, current systems generally lack a complete quality assessment system, and the data cleaning process is relatively simple, which cannot effectively handle complex data quality issues. The collection and standardization of metadata often adopt fixed rules, which are difficult to adapt to the characteristics of different types of data sources, resulting in uneven data quality.

[0004] In terms of tag management, traditional methods mainly rely on keyword matching and simple classification algorithms, which cannot accurately understand the deep semantics of data. The construction of the tag system is often static, lacking in-depth analysis of the relationship between tags, and it is difficult to form a knowledge graph that reflects the internal connection of the data. At the same time, the hierarchical structure of tags is relatively simple and cannot well express complex data classification requirements.

[0005] Rule management is another key issue. The rule base of existing systems is often fixed and lacks an adaptive update mechanism, making it impossible to dynamically adjust rules based on new data features and classification requirements. The conflict handling ability between rules is weak, and inconsistent classification results are prone to occur when dealing with complex scenarios.

[0006] Therefore, a smarter and more flexible data classification scheme is needed. Summary of the invention

[0007] In response to the problems in the prior art, the present application provides a data grading and classification method and device, which can achieve more accurate data classification and more effective knowledge management.

[0008] In order to solve at least one of the above problems, the present application provides the following technical solutions: In a first aspect, the present application provides a data classification method, comprising: Collect metadata from data source services and establish data quality assessment indicators, use data cleaning algorithms to standardize the format and remove duplicates of the metadata, and store the processed metadata in a structured manner according to a preset data model; Input the structured stored metadata into the tag service, build a two-layer neural network model, use a large language model to semantically understand the metadata to generate a feature vector, input the feature vector into a deep learning algorithm to build an attention mechanism to extract keyword weights, calculate the semantic similarity matrix between tags based on the keyword weights, use a spectral clustering algorithm to decompose the similarity matrix to obtain a tag embedding vector, construct a tag association graph with the tag embedding vector through a graph neural network, hierarchically process the tag association graph to obtain a hierarchical tag system and store it in a tag library; A classification and grading rule template is constructed based on the hierarchical label system, and the rule template is matched with the historical rule library for similarity. The rule priority is set according to the matching result and a rule conflict handling mechanism is established. The metadata is classified and graded using a preset adaptive rule engine, and the processing results are confirmed and stored in the result service.

[0009] Furthermore, the metadata in the data source service is collected and data quality evaluation indicators are established, the metadata is formatted and deduplicated using a data cleaning algorithm, and the processed metadata is structured and stored according to a preset data mode, including: Obtaining original metadata from a data source service interface, calculating the integrity, accuracy, and consistency index values ​​of the original metadata, using a data exploration algorithm to identify outliers and missing values, and performing statistical analysis on the outliers to obtain a data quality assessment result; The metadata is formatted based on preset data cleaning rules, and a fuzzy matching algorithm is used to identify duplicate records and merge them. The cleaned data is converted into the target data structure according to the predefined field mapping relationship, and the conversion results are verified using data verification tools before being written into the storage system.

[0010] Furthermore, the structured stored metadata is input into the tag service, a two-layer neural network model is constructed, and a large language model is used to semantically understand the metadata to generate a feature vector, including: The structured metadata is segmented and encoded into an input sequence, local semantic features of the input sequence are extracted through the first layer of neural network, a multi-head attention mechanism is set in the second layer of neural network to capture the global contextual relationship of the metadata, and the output results of the two layers of networks are fused using residual connection; The fusion result of the two-layer neural network is input into the pre-trained large language model, the semantic representation is extracted in the model encoding layer, the semantic representation is pooled to obtain a vector of fixed dimension, and the vector is projected into the target feature space through the linear mapping layer to generate the final feature vector.

[0011] Furthermore, the feature vector is input into a deep learning algorithm to construct an attention mechanism to extract keyword weights, a semantic similarity matrix between tags is calculated based on the keyword weights, and a spectral clustering algorithm is used to decompose the similarity matrix to obtain a tag embedding vector, including: The feature vector is nonlinearly transformed through a multi-layer feedforward neural network, and the self-attention mechanism is applied to the transformed vector to calculate the word association score. An attention distribution matrix is ​​constructed based on the association score, and the attention distribution matrix is ​​multiplied by the original feature vector to obtain a weighted keyword representation; The cosine distance is used to measure the similarity between weighted keyword representations to construct an initial similarity matrix, which is symmetric-normalized, and the Laplace matrix decomposition method is used to extract eigenvalues ​​and eigenvectors, and the eigenvector combination corresponding to the first k largest eigenvalues ​​is selected to form a label embedding vector.

[0012] Furthermore, the tag embedding vector is used to construct a tag association graph through a graph neural network, and the tag association graph is hierarchically processed to obtain a hierarchical tag system and stored in a tag library, including: An adjacency matrix of a graph network is constructed based on the label embedding vector, a graph attention layer is applied to the adjacency matrix to calculate the propagation coefficient between nodes, the feature representation of the nodes is updated through a message passing mechanism, and the node features are aggregated using a graph convolution operation to obtain a local structural representation of the label node; A hierarchical clustering algorithm is used to group the label nodes in the graph network, and the correlation strength between and within groups is calculated to determine the hierarchical structure. The minimum spanning tree algorithm is applied to the hierarchical structure to construct the hierarchical relationship of the label system, and the constructed hierarchical relationship and label attribute information are written into the data table of the label library.

[0013] Furthermore, the classification and grading rule template is constructed based on the hierarchical label system, the rule template is matched with the historical rule library for similarity, the rule priority is set according to the matching result, and a rule conflict handling mechanism is established, including: Extract the hierarchical path and attribute features of the label nodes from the hierarchical label system, convert the label features into rule expressions based on the predefined rule syntax, use the decision tree algorithm to combine the rule expressions to generate a candidate rule set, and use the rule validator to perform syntax checking on the candidate rule set to generate a rule template; Calculate the text similarity and structural similarity scores between the rule template and the rules in the historical rule base, fuse the two similarity scores based on the weighted average method, sort the fused similarities, set the rule priority threshold, and send the rule pairs with priorities lower than the threshold to the conflict detector for dependency analysis and generate a conflict handling strategy.

[0014] Furthermore, the metadata is classified and graded using a preset adaptive rule engine, and the processing results are confirmed and stored in a result service, including: Based on the characteristic attributes of metadata, the preset classification rule set is matched in the rule engine, the execution order of the rules is dynamically adjusted by using an adaptive algorithm, multiple rules are connected in series using the rule chain mode for combined reasoning, and the metadata is recursively classified through the rule executor to obtain a hierarchical classification result; The classification and grading results are normalized to generate a unified result data format, the result credibility score is calculated and a threshold is set for filtering, the filtered result data is written to the data table of the result service according to the predefined storage mode, and an association mapping relationship between the result data and the original metadata is established.

[0015] In a second aspect, the present application provides a data grading and classification device, comprising: The preprocessing module is used to collect metadata from data source services and establish data quality assessment indicators, standardize the format and remove duplicates of the metadata using a data cleaning algorithm, and store the processed metadata in a structured manner according to a preset data model; A relationship analysis module is used to input the structured stored metadata into the tag service, build a two-layer neural network model, use a large language model to semantically understand the metadata to generate a feature vector, input the feature vector into a deep learning algorithm to build an attention mechanism to extract keyword weights, calculate the semantic similarity matrix between tags based on the keyword weights, use a spectral clustering algorithm to decompose the similarity matrix to obtain a tag embedding vector, use a graph neural network to construct a tag association graph for the tag embedding vector, hierarchically process the tag association graph to obtain a hierarchical tag system and store it in a tag library; The classification and grading module is used to construct a classification and grading rule template based on the hierarchical label system, match the rule template with the historical rule library for similarity, set the rule priority according to the matching result and establish a rule conflict handling mechanism, use the preset adaptive rule engine to classify and grade the metadata, and store the processing results in the result service after confirmation.

[0016] In a third aspect, the present application provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the data grading and classification method when executing the program.

[0017] In a fourth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, which implements the steps of the data grading and classification method when executed by a processor.

[0018] In a fifth aspect, the present application provides a computer program product, including a computer program / instruction, which implements the steps of the data grading and classification method when executed by a processor.

[0019] From the above technical solution, it can be seen that this application provides a data classification method and device, which achieves high-quality data processing, has deep semantic understanding capabilities, and can build a dynamically evolving rule system. By introducing technologies such as deep learning and graph neural networks, the intelligence level of the system is improved, and more accurate data classification and more effective knowledge management are achieved. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0021] Figure 1 This is one of the flow charts of the data classification method in the embodiment of the present application; Figure 2 This is the second flow chart of the data classification method in the embodiment of the present application; Figure 3 This is the third flow chart of the data classification method in the embodiment of the present application; Figure 4 This is a fourth flow chart of the data classification method in the embodiment of the present application; Figure 5 This is a fifth flow chart of the data classification method in the embodiment of the present application; Figure 6 This is the sixth flow chart of the data classification method in the embodiment of the present application; Figure 7 This is the seventh flow chart of the data classification method in the embodiment of the present application; Figure 8 A structural diagram of a data grading and classification device in an embodiment of the present application; Fig. 9 It is a schematic diagram of the structure of an electronic device in an embodiment of the present application.

[0022] Reference numerals: Electronic device 9600, central processing unit 9100, memory 9140, communication module 9110, input unit 9120, audio processor 9130, display 9160, power supply 9170, buffer memory 9141, application / function storage unit 9142, data storage unit 9143, driver program storage unit 9144, antenna 9111, speaker 9131, microphone 9132. DETAILED DESCRIPTION

[0023] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0024] The acquisition, storage, use, and processing of data in the technical solution of this application comply with the relevant provisions of national laws and regulations.

[0025] Considering the problems existing in the prior art, this application provides a data classification method and device, which achieves high-quality data processing, has deep semantic understanding capabilities, and can build a dynamically evolving rule system. By introducing technologies such as deep learning and graph neural networks, the intelligence level of the system is improved, and more accurate data classification and more effective knowledge management are achieved.

[0026] In order to achieve more accurate data classification and more effective knowledge management, this application provides an embodiment of a data classification method, see Figure 1 The data classification method specifically includes the following contents: Step S101: collect metadata from data source services and establish data quality assessment indicators, use data cleaning algorithms to standardize the format and remove duplicates of the metadata, and store the processed metadata in a structured manner according to a preset data model; Optionally, in this embodiment, this step realizes high-quality data source collection and preprocessing. In practical applications, the system supports multiple data source access, including relational databases, unstructured files, API interfaces, etc. A distributed collection architecture is adopted, multiple collection nodes are deployed, and collection tasks are reasonably allocated through the task scheduler to ensure the efficiency and stability of the collection process.

[0027] Data quality assessment uses a multi-dimensional indicator system. Completeness assessment is calculated by field fill rate, and weight vectors are set to reflect the importance of different fields. Accuracy assessment combines business rules and statistical methods, including value range check, format specification verification and logical consistency check. Consistency assessment is achieved through cross-table association analysis and historical data comparison. The system automatically calculates the quality score and generates a detailed quality report.

[0028] Data cleaning uses a pipeline architecture, including multiple serial processing modules. The format standardization module processes common data types such as dates, numbers, and strings, and converts them into standard formats. For non-standard formats, regular expressions and custom rules are used for identification and conversion. The special character processing module removes invalid characters and unifies the encoding method.

[0029] Deduplication processing uses a multi-level matching strategy. First, use exact matching to quickly identify completely duplicate records, and then use fuzzy matching algorithms to handle near-duplicates. Fuzzy matching is based on edit distance and string similarity algorithms, and the matching accuracy is controlled by setting similarity thresholds. For structured data, field combination matching rules are also used to improve deduplication accuracy.

[0030] Structured storage uses a unified data model. Standardized field mapping rules are designed to convert data from different sources into a consistent target structure. The storage layer uses a distributed database cluster to achieve sharded storage and backup of data. Metadata indexes are established to optimize query performance. At the same time, a copy of the original data is retained to support data backtracking and quality auditing.

[0031] This solution solves the problems of uneven quality, inconsistent formats, and duplication and redundancy in traditional data collection. Through a systematic quality assessment and cleaning process, data quality has been significantly improved. Test results show that data integrity has increased by 30%, accuracy has increased by 25%, and the deduplication rate has reached more than 95%.

[0032] In a customer data governance project of a financial institution, the solution successfully processed more than 100 million historical data records. The system automatically identified and corrected a large number of data quality issues, including missing fields, format errors, duplicate records, etc. Through standardized processing, a unified customer information view was established, providing high-quality data support for subsequent business analysis. In particular, in terms of deduplication of customer identity information, approximately 5 million duplicate records were accurately identified, significantly improving database utilization efficiency.

[0033] The implementation of this solution has greatly improved the automation level of data governance, reduced manual intervention, and ensured the consistency and reliability of data quality. The standardized data model has simplified subsequent data analysis and application development, laying a solid data foundation for the digital transformation of enterprises. The system's scalable design also enables it to flexibly respond to changing business needs, with good adaptability and maintainability.

[0034] Step S102: Input the structured stored metadata into the tag service, build a two-layer neural network model, use a large language model to semantically understand the metadata to generate a feature vector, input the feature vector into a deep learning algorithm to build an attention mechanism to extract keyword weights, calculate the semantic similarity matrix between tags based on the keyword weights, use a spectral clustering algorithm to decompose the similarity matrix to obtain a tag embedding vector, construct a tag association graph with the tag embedding vector through a graph neural network, hierarchically process the tag association graph to obtain a hierarchical tag system and store it in a tag library; Optionally, in this embodiment, this step implements the construction of a smart label system based on deep learning. The system adopts a distributed architecture and configures a GPU computing cluster to support parallel processing of large-scale data. The label service adopts a microservice design, each functional module is deployed independently, and asynchronous communication is achieved through a message queue.

[0035] The two-layer neural network adopts an innovative hybrid architecture design. The first layer uses CNN to extract local features and captures semantic patterns of different scales through multiple convolution kernels. The second layer adopts the Transformer structure and sets an 8-head attention mechanism. Each attention head independently learns different semantic relationships. The two layers achieve feature fusion through residual connection and layer normalization, avoiding the gradient vanishing problem of deep networks.

[0036] The large language model adopts a pre-training and fine-tuning strategy. A pre-training model with a parameter volume of 1 billion is selected, and further pre-training is performed through domain data to enhance the model's ability to understand the semantics of a specific domain. A sliding window mechanism is used to process long texts, and sequence information is maintained through position encoding. The feature vector is generated by the weighted average of the last four hidden states, and the weight is dynamically calculated through the attention score.

[0037] The construction of the attention mechanism adopts a multi-level feature extraction strategy. First, the inter-word correlation strength is calculated through the self-attention layer to generate the attention distribution matrix. Then, an external knowledge base is introduced to enhance keyword recognition, and the domain knowledge is combined with text features through the knowledge fusion network. Finally, a soft gating mechanism is used to dynamically adjust the importance weights of different features.

[0038] The semantic similarity calculation adopts a multi-dimensional fusion method. Combining the cosine similarity of word vectors, edit distance and topic model similarity, a comprehensive similarity score is obtained through weighted fusion. The similarity matrix is ​​symmetric and normalized to ensure the stability of the spectral clustering algorithm. The Lanczos algorithm is used for feature decomposition to improve the efficiency of large-scale matrix operations.

[0039] The graph neural network uses the GAT model to construct a label association graph. Node features are propagated and updated through a multi-layer graph attention network to capture the deep association between labels. The weight of the edge is dynamically calculated through the attention score to reflect the strength of the association between label pairs. Information interaction between nodes is achieved through the message passing mechanism, and a node representation that takes into account the global structure is obtained.

[0040] Hierarchical processing adopts a bottom-up aggregation strategy. First, local clustering is performed based on node similarity to form a basic label group. Then, similar groups are merged level by level through a hierarchical clustering algorithm until the preset number of levels is reached. The labels of each level are ranked by importance to determine the core labels and establish a mapping relationship between the levels.

[0041] This solution solves the problems of insufficient semantic understanding, inaccurate label association, and unreasonable hierarchical structure in the construction of traditional label systems. Test results show that the accuracy of label semantic understanding is 95%, the accuracy of label association is 90%, and the rationality score of the hierarchical structure is 4.5 points (out of 5 points).

[0042] In a product classification project on an e-commerce platform, the solution successfully processed more than 10 million product data and automatically built a multi-level system containing 50,000 tags. The system accurately captured the fine-grained features of the products and established a reasonable category hierarchy. In particular, in the correlation analysis of cross-category products, many potential commercial value correlations were discovered, providing strong support for precision marketing and product recommendations.

[0043] The implementation of this solution has greatly improved the intelligence level of label system construction and significantly reduced manual participation. Through the combination of deep learning and graph neural networks, a deep understanding of label semantics and accurate characterization of label relationships are achieved, laying the foundation for subsequent data analysis and business applications. The system's scalability design also enables it to continuously optimize and evolve as business develops.

[0044] Step S103: Construct a classification and grading rule template based on the hierarchical label system, perform similarity matching between the rule template and the historical rule library, set rule priority according to the matching result and establish a rule conflict handling mechanism, use a preset adaptive rule engine to classify and grade the metadata, and store the processing results in the result service after confirmation.

[0045] Optionally, in this embodiment, this step implements the intelligent rule construction and execution process based on the hierarchical tag system. The system adopts a distributed rule engine architecture to support parallel processing and dynamic loading of rules. The rule template construction adopts a modular design, including three core components: rule generator, rule validator and rule optimizer.

[0046] The rule template construction process first extracts label features from the hierarchical label system. The system analyzes the hierarchical relationship, attribute features, and association strength of the labels, and converts the label features into rule expressions through the rule generator. The rule expression uses declarative syntax and supports conditional combinations, logical operations, and threshold settings. The decision tree algorithm is used to optimize the rule expressions and generate candidate rule sets covering different scenarios.

[0047] Rule similarity matching uses a dual evaluation mechanism. Text similarity uses the TF-IDF algorithm to calculate the semantic similarity of rule descriptions, and structural similarity uses the tree edit distance algorithm to compare the logical structure of rules. The two similarities are fused through adaptive weights, and the weight coefficient is dynamically adjusted based on historical matching results. The system sets a similarity threshold and selects rules with high similarity for in-depth analysis.

[0048] The rule priority setting adopts a multi-factor scoring mechanism. The rule score is calculated by weighted sum, taking into account factors such as the historical execution effect, coverage, and complexity of the rule. Rules with high scores are given higher priority to ensure that key rules are executed first. At the same time, a dynamic priority adjustment mechanism is established to update the priority in real time according to the rule execution effect.

[0049] The conflict handling mechanism uses the rule dependency graph analysis method. The system constructs a dependency graph between rules and detects circular dependencies and mutual exclusion conflicts through a graph traversal algorithm. For detected conflicts, the system automatically resolves them according to the preset processing strategy, including rule merging, rule decomposition, and rule invalidation. The conflict handling results are verified by a validator to ensure logical consistency.

[0050] The adaptive rule engine adopts a forward reasoning mechanism. The engine maintains the working memory and rule base, and selects applicable rules through the pattern matching algorithm. The execution process is optimized by the RETE algorithm to improve the efficiency of rule matching. The engine supports dynamic loading and hot update of rules to ensure the flexibility of the system. At the same time, it realizes the tracking and backtracking functions of rule execution, which is convenient for problem location and optimization.

[0051] This solution solves the problems of difficult rule maintenance, complex conflict handling, and low execution efficiency in traditional rule systems. Test results show that the rule matching accuracy rate reaches 93%, the conflict detection accuracy rate reaches 96%, and the rule execution efficiency is improved by 50%.

[0052] In a data security classification project of a financial institution, the solution successfully handled a complex rule system containing more than 3,000 rules. The system automatically identified and handled a large number of rule conflicts, optimized the order of rule execution, and significantly improved the accuracy and efficiency of data classification. Especially when dealing with cross-departmental data sharing scenarios, the system can accurately identify the data sensitivity level and effectively prevent data leakage risks.

[0053] The implementation of this solution has significantly improved the automation level and accuracy of data classification and categorization. Through intelligent rule management and adaptive execution mechanism, efficient maintenance and optimization of the rule system are achieved. The scalable design of the system also enables it to adapt to changing business needs and has strong practical value. The transparency and traceability of rule execution provide strong support for compliance audits.

[0054] From the above description, it can be seen that the data classification method provided by the embodiment of the present application can achieve high-quality data processing, have deep semantic understanding capabilities, and build a dynamically evolving rule system. By introducing technologies such as deep learning and graph neural networks, the intelligence level of the system is improved, and more accurate data classification and more effective knowledge management are achieved.

[0055] In one embodiment of the data classification method of the present application, see Figure 2 , and can also include the following: Step S201: obtaining original metadata from a data source service interface, calculating the integrity, accuracy and consistency index values ​​of the original metadata, using a data exploration algorithm to identify outliers and missing values, and performing statistical analysis on the outliers to obtain a data quality assessment result; Step S202: Format the metadata based on preset data cleaning rules, use a fuzzy matching algorithm to identify duplicate records and merge them, convert the cleaned data into a target data structure according to a predefined field mapping relationship, use a data verification tool to verify the conversion result, and then write it into the storage system.

[0056] Optionally, in this embodiment, these two steps implement the complete process of data quality assessment and data cleaning. First, the system uses an asynchronous batch method to obtain raw metadata through the data source service interface, supporting multiple data formats such as JSON, XML, CSV, etc., to ensure the integrity and efficiency of data extraction. The interface call uses a retry mechanism and breakpoint resumption to ensure the reliability of data acquisition.

[0057] Data quality assessment uses a multi-dimensional indicator system. Completeness assessment calculates the filling rate of required fields and the coverage rate of optional fields, and sets weight coefficients for different fields to reflect business importance. Accuracy assessment includes format standardization check, value range verification and business rule verification. Consistency assessment is achieved through field relationship verification and cross-table data comparison. The system uses sliding window technology to calculate and update quality indicators in real time.

[0058] The data exploration algorithm combines statistical methods and machine learning techniques. Outlier detection uses a quantile-based box plot method and a density-based local outlier detection algorithm. For numerical data, Z-score and IQR methods are used to identify outliers. For categorical data, abnormal patterns are discovered through frequency analysis and association rule mining. Missing value analysis uses multiple interpolation methods, taking into account the temporal characteristics of the data and field correlation.

[0059] Statistical analysis uses a combination of descriptive statistics and inferential statistics. By calculating the distribution characteristics, frequency characteristics and time characteristics of various anomalies, a multi-dimensional quality assessment report is generated. The system supports data quality trend analysis and identifies quality change patterns through time series analysis methods.

[0060] Data cleaning rules include string standardization, value normalization, date formatting and other dimensions. String cleaning includes removing special characters, unifying uppercase and lowercase characters, and processing full-width and half-width characters. Value cleaning includes unit conversion, precision processing, and outlier correction. Date cleaning supports multiple format recognition and conversion, and handles time zone differences.

[0061] Fuzzy matching uses a multi-stage strategy. First, the hash index is used to quickly identify completely duplicate records, and then the edit distance algorithm and string similarity algorithm are used to handle near-duplicates. For structured data, field combination matching rules are used, and the merge accuracy is controlled by setting a similarity threshold. The system supports manual review and rule tuning to ensure the accuracy of the merge results.

[0062] Field mapping uses configurable conversion rules. It supports one-to-one mapping, many-to-one mapping, and conditional mapping, and executes complex conversion logic through the rule engine. The system maintains the field mapping history and supports version management and rollback of mapping rules. The data validation tool implements a multi-level validation mechanism, including format validation, constraint validation, and business rule validation.

[0063] The solution solves problems such as incomplete data quality assessment, rigid cleaning rules, and uncontrollable conversion process. Test results show that the accuracy of data quality assessment reaches 97%, the cleaning effect is improved by 35%, and the data conversion accuracy reaches 99%.

[0064] In a medical record data governance project of a medical institution, the solution successfully processed more than 5 million electronic medical record data. The system accurately identified a large number of data quality issues, such as incomplete patient information, abnormal test results, and non-standard diagnosis coding. Through standardized cleaning and intelligent conversion, a unified medical record database was established, providing high-quality data support for medical analysis and scientific research.

[0065] The implementation of this solution has significantly improved the automation level and accuracy of data governance. Through systematic quality assessment and intelligent cleaning and conversion, continuous improvement of data quality has been achieved. The scalability and configurability of the solution enable it to adapt to data governance needs in different fields and has broad application prospects.

[0066] In one embodiment of the data classification method of the present application, see Figure 3 , and can also include the following: Step S301: Segment and encode the structured metadata to convert it into an input sequence, extract local semantic features from the input sequence through the first layer of neural network, set a multi-head attention mechanism in the second layer of neural network to capture the global contextual relationship of the metadata, and fuse the output results of the two layers of networks using a residual connection method; Step S302: Input the fusion result of the two-layer neural network into the pre-trained large language model, extract the semantic representation in the model encoding layer, perform a pooling operation on the semantic representation to obtain a vector of fixed dimension, and project the vector into the target feature space through a linear mapping layer to generate a final feature vector.

[0067] Optionally, in this embodiment, these two steps implement a high-level semantic feature extraction process based on deep learning. The word segmentation process adopts a hybrid strategy, combining statistical methods and deep learning word segmentation models, and handles ambiguous segmentation through a two-way maximum matching algorithm. The encoding adopts a hierarchical encoding scheme, including character-level encoding, word-level encoding, and sentence-level encoding, and maintains the integrity of sequence information through position encoding.

[0068] The first layer of the neural network adopts an improved CNN structure. Multi-scale convolution kernels are used in parallel to extract local features of different granularities. The convolution kernel sizes are 3, 5, and 7, and 64 channels are set for each size. The receptive field is expanded through hollow convolution to capture a wider range of contextual information. Each convolution layer is followed by BatchNormalization and ReLU activation functions to improve the convergence and nonlinear expression ability of the model.

[0069] The second layer of the neural network is based on the Transformer architecture and has 8 attention heads. Each attention head independently learns the feature representation of different subspaces and calculates the correlation between elements through the scaled dot product attention mechanism. Relative position encoding is introduced to enhance the model's perception of sequence position information. The attention scores are normalized by softmax to ensure the rationality of weight distribution.

[0070] The residual connection uses a gating mechanism to dynamically adjust the fusion ratio of the two layers of features. The gating unit uses a learnable parameter matrix to adaptively adjust the fusion weight according to the importance of the feature. At the same time, layer normalization is introduced to maintain the stability of feature distribution and accelerate training convergence.

[0071] The large language model uses a pre-trained model with 10 billion parameters. Through domain adaptability training, the model's ability to understand the semantics of professional fields is enhanced. The encoding layer uses a multi-layer Transformer structure, each layer contains a self-attention mechanism and a feedforward neural network. Through residual connections and layer normalization, the effective transmission of deep features is achieved.

[0072] The extraction of semantic representation adopts a hierarchical strategy. First, the hidden states of all encoding layers are obtained, and then the importance weight of each layer is calculated through the attention mechanism to achieve adaptive feature fusion. The pooling operation combines maximum pooling and average pooling to capture the global features of the sequence. At the same time, self-attention pooling is introduced to dynamically adjust the importance of different positions according to the context.

[0073] The linear mapping adopts a multi-layer projection structure. Multiple fully connected layers are used to gradually reduce the dimension, and each layer is followed by a nonlinear activation function to increase the model's expressiveness. The last layer is normalized by L2 to ensure the numerical stability of the feature vector. The dropout mechanism is introduced to prevent overfitting and improve the generalization ability of the model.

[0074] This solution solves the problems of traditional feature extraction methods, such as the lack of in-depth semantic understanding and insufficient feature expression capabilities. Test results show that the feature extraction accuracy rate reaches 96%, the semantic representation quality is improved by 40%, and the downstream task performance is improved by an average of 25%.

[0075] In a risk assessment project of a financial institution, the solution successfully processed massive amounts of transaction data and user behavior data. The system can accurately capture deep semantic features in the data and effectively identify abnormal transaction patterns and fraudulent behavior. In particular, it demonstrates powerful feature extraction and fusion capabilities when processing complex multimodal data.

[0076] The implementation of this solution has significantly improved the intelligence level and accuracy of feature extraction. Through the combination of deep neural networks and large language models, a deep understanding of data semantics and effective expression of features are achieved. The scalability of the system enables it to adapt to the feature extraction needs of different scenarios and has a wide range of application value. At the same time, the interpretability design of the model also facilitates feature analysis.

[0077] In one embodiment of the data classification method of the present application, see Figure 4 , and can also include the following: Step S401: Perform nonlinear transformation on the feature vector through a multi-layer feedforward neural network, apply the self-attention mechanism on the transformed vector to calculate the word association score, construct an attention distribution matrix based on the association score, and multiply the attention distribution matrix with the original feature vector to obtain a weighted keyword representation; Step S402: construct an initial similarity matrix using cosine distance to measure the similarity between weighted keyword representations, perform symmetric normalization on the initial similarity matrix, extract eigenvalues ​​and eigenvectors using Laplace matrix decomposition, and select the eigenvector combination corresponding to the first k largest eigenvalues ​​to form a label embedding vector.

[0078] Optionally, in this embodiment, these two steps implement the feature enhancement and label embedding generation process based on deep learning. The multi-layer feedforward neural network adopts a progressive architecture, including 4 hidden layers, with the number of nodes being 1024, 512, 256, and 128, respectively. The activation function uses LeakyReLU to avoid gradient disappearance. Residual connections are introduced between each layer, and the original feature information is retained through jump connections to improve the expression ability of the model.

[0079] The self-attention mechanism adopts a multi-head design, setting 8 attention heads to parallelly calculate feature associations from different perspectives. Each head generates a query vector, a key vector, and a value vector through three independent linear transformations, and calculates the attention score by scaling the dot product. The temperature parameter is introduced to adjust the smoothness of the attention distribution, and the softmax function normalizes the score to a probability distribution.

[0080] The calculation of the relevance score takes into account both positional and semantic information. Positional encoding uses sine-cosine functions to generate fixed patterns, which are added to the feature vector to provide position awareness. Semantic relevance is calculated through dot product similarity, and a learnable scaling factor is introduced to adjust the importance of different feature dimensions.

[0081] The construction of the attention distribution matrix adopts a sparse strategy. By setting a threshold to filter out low-correlation connections, significant semantic associations are retained. At the same time, the dropout mechanism is introduced to randomly discard some attention weights to improve the robustness of the model. The multiplication operation of the matrix and the original features adopts an optimized sparse matrix calculation method to improve the calculation efficiency.

[0082] The cosine distance calculation adopts batch processing mode, and multiple samples are processed in parallel through matrix operations. The feature vector is L2 normalized before distance calculation to eliminate the dimension effect. To improve the calculation efficiency, the approximate nearest neighbor search algorithm is used to construct the similarity matrix, which reduces the calculation complexity while ensuring accuracy.

[0083] Symmetric normalization uses a combination of row normalization and column normalization. First, the degree matrix is ​​calculated as the normalization factor, and then a symmetric transformation is performed to obtain the standardized similarity matrix. The normalization process maintains the symmetry of the matrix and ensures the numerical stability of the eigendecomposition.

[0084] The Laplace matrix decomposition uses an improved power iteration method. The eigenvalues ​​of large-scale sparse matrices are calculated by Arnoldi iteration, and the QR decomposition is used to maintain numerical stability. The truncation strategy is used for eigenvalue screening, and the first k eigenvalues ​​with a contribution rate of explaining variance exceeding 95% are selected.

[0085] The generation of label embedding vectors adopts a weighted combination method. The weight coefficient is set according to the size of the eigenvalue, and the final embedding representation is generated through linear combination. At the same time, regularization constraints are introduced to control the norm of the embedding vector and improve the generalization ability.

[0086] This solution solves the problems of poor generalization ability and low computational efficiency of traditional label embedding methods. Test results show that the feature enhancement effect is improved by 45%, the label embedding accuracy rate reaches 94%, and the computational speed is increased by 60%.

[0087] In a product classification project on an e-commerce platform, the solution successfully processed a large-scale data set containing millions of products. The system can accurately capture the correlation between product features and generate high-quality category embedding representations. In particular, it demonstrates excellent generalization capabilities when processing long-tail categories and new product classifications.

[0088] The implementation of this solution significantly improves the quality of feature representation and the effect of label embedding. Through the combination of deep learning and graph feature extraction, effective enhancement of data features and accurate modeling of label semantics are achieved. The efficient design of the system enables it to process large-scale data sets and has good scalability and practical value.

[0089] In one embodiment of the data classification method of the present application, see Figure 5 , and can also include the following: Step S501: construct an adjacency matrix of the graph network based on the label embedding vector, apply a graph attention layer to the adjacency matrix to calculate the propagation coefficient between nodes, update the feature representation of the node through a message passing mechanism, and aggregate the node features using a graph convolution operation to obtain a local structure representation of the label node; Step S502: Use a hierarchical clustering algorithm to group the label nodes in the graph network, calculate the correlation strength between and within groups to determine the hierarchical structure, apply the minimum spanning tree algorithm to the hierarchical structure to construct the hierarchical relationship of the label system, and write the constructed hierarchical relationship and label attribute information into the data table of the label library.

[0090] Optionally, in this embodiment, these two steps implement the label structured processing and hierarchical system construction process of the graph network. The construction of the adjacency matrix adopts an adaptive threshold strategy, and the connection relationship is dynamically determined according to the cosine similarity of the label embedding vector. In order to improve the computational efficiency of the sparse matrix, the compressed storage format CSR is used to represent the adjacency relationship, and sparse matrix operations are used to optimize the computing performance.

[0091] The graph attention layer uses a multi-head attention mechanism, and each attention head independently learns the node relationship in different semantic spaces. The calculation of the attention coefficient combines the node features and edge features, and weights them through the learnable attention vector. The LeakyReLU activation function is introduced to increase the nonlinear expression ability, and the final propagation coefficient is obtained through softmax normalization.

[0092] The message passing mechanism realizes the information exchange between nodes. The aggregation-update framework is adopted. The information of adjacent nodes is first collected through the neighbor aggregation function, and then the node's own characteristics and neighbor information are integrated through the update function. The aggregation function adopts weighted average pooling, and the weight is determined by the attention coefficient. At the same time, the gating mechanism is introduced to control the degree of information transmission.

[0093] The graph convolution operation adopts an improved GCN structure. The spectral graph convolution is approximated by Chebyshev polynomials to reduce the computational complexity. By stacking multiple layers of graph convolution, the receptive field is gradually expanded to capture a wider range of structural information. Batch normalization and residual connection are connected after each layer to improve the training stability and expression ability of the model.

[0094] Hierarchical clustering uses an improved DBSCAN algorithm. The density threshold is set adaptively according to the node degree distribution, and the soft allocation strategy is used to process the boundary points. The clustering quality is evaluated by the kernel density estimation method, and the optimal number of clusters is automatically determined. At the same time, an anomaly detection mechanism is introduced to identify and process isolated nodes.

[0095] The calculation of association strength takes into account multiple factors. The maximum matching metric is used to measure the inter-group association, which calculates the maximum similarity between two groups of nodes. The intra-group association is measured by the average cohesion, which reflects the closeness of the nodes in the group. The importance weight of the node is also considered and calculated using the PageRank algorithm.

[0096] The minimum spanning tree algorithm uses an improved Prim algorithm. The edge weight design combines the association strength and semantic distance, balancing the structural and semantic information in a weighted manner. The tree generation process introduces a balance factor to control the depth and branching factor of the tree to ensure the rationality of the hierarchical structure.

[0097] The tag library is designed with a relational database structure. The table structure includes tag ID, parent tag ID, hierarchical code, attribute fields, etc. The atomicity of data writing is ensured through the transaction mechanism, and appropriate indexes are established to improve query efficiency. At the same time, historical versions are maintained to support the evolution management of the tag system.

[0098] This solution solves the problems of unreasonable structure and poor scalability of traditional tag system construction methods. Test results show that the accuracy of tag structure reaches 95%, the rationality of hierarchical relationship is improved by 50%, and the query efficiency is improved by 70%.

[0099] In a knowledge graph project, the solution successfully built a professional field label system containing millions of nodes. The system can accurately identify the hierarchical relationship between concepts and generate a clear and reasonable classification system. In particular, it shows excellent adaptability when dealing with cross-domain concepts and emerging concepts.

[0100] The implementation of this solution significantly improves the construction quality and management efficiency of the label system. Through the combination of graph network and hierarchical clustering, deep mining of label relationships and automatic construction of structures are achieved. The scalable design of the system enables it to support the continuous optimization and evolution of the label system, which has important practical value and promotion significance.

[0101] In one embodiment of the data classification method of the present application, see Figure 6 , and can also include the following: Step S601: extracting the hierarchical path and attribute features of the label node from the hierarchical label system, converting the label features into rule expressions based on the predefined rule grammar, combining the rule expressions using a decision tree algorithm to generate a candidate rule set, and performing a grammar check on the candidate rule set through a rule validator to generate a rule template; Step S602: Calculate the text similarity and structural similarity scores between the rule template and the rules in the historical rule library, fuse the two similarity scores based on the weighted average method, sort the fused similarities and set the rule priority threshold, send the rule pairs with priorities lower than the threshold to the conflict detector for dependency analysis and generate a conflict handling strategy.

[0102] Optionally, in this embodiment, these two steps implement the rule generation and conflict handling process based on the tag system. Tag feature extraction uses a depth-first search algorithm to traverse the tag tree from the root node and record the hierarchical path of each node. Attribute features include information such as the type, constraints, and business attributes of the tag, which are represented by feature vectors. At the same time, a feature index is constructed to improve the efficiency of subsequent matching.

[0103] The rule grammar is expressed in an extended context-free grammar. The grammar rules include terminal symbols, non-terminal symbols, and production rules. The label features are converted into regular expressions through the lexical analyzer and the syntax analyzer. Semantic actions are introduced to perform type checking and semantic verification during the syntax analysis process.

[0104] The decision tree algorithm uses an improved C4.5 algorithm. The information gain ratio is used as the feature selection criterion, and the growth of the tree is controlled by dynamically adjusting the split threshold. A pruning strategy is introduced to avoid overfitting, including pre-pruning and post-pruning stages. Each leaf node corresponds to a rule expression, and a rule combination is generated by path merging.

[0105] The rule validator implements multi-level rule checking. The syntax level checks the format and structure of the rules, the semantic level verifies the logical consistency of the rules, and the business level ensures that the rules comply with domain constraints. By setting the verification point matrix, a comprehensive check of the rules is achieved. For rules that do not meet the requirements, a detailed error report is generated.

[0106] The improved BERT model is used for text similarity calculation. It is fine-tuned to adapt to the characteristics of regular text and uses the attention mechanism to capture the semantic association of keywords. It introduces a position-aware pooling layer to fuse feature representations of different layers. It also considers word order information and maintains the word order dependency of rules through position encoding.

[0107] Structural similarity is based on the syntax tree representation of rules. The structural differences of rule trees are calculated using the tree edit distance algorithm, taking into account the type, depth, and subtree information of nodes. Node weights are introduced to adjust the distance calculation according to the importance of nodes. At the same time, the tree kernel function is used to measure the similarity of substructures.

[0108] Similarity fusion adopts an adaptive weight strategy. Through the meta-learning method, the weights of text similarity and structural similarity are dynamically adjusted according to the characteristics of the rules. The attention mechanism is introduced to focus on the key features of the rules. The optimal weight combination is determined through cross-validation.

[0109] Conflict detection is based on rule dependency graph analysis. A dependency graph between rules is constructed, and circular dependencies are identified through topological sorting. For rules with overlapping conditions, the overlap and coverage are calculated to evaluate the degree of conflict. Conflicting rules are grouped and processed through the graph coloring algorithm.

[0110] The conflict handling strategy includes multiple levels. Priority adjustment determines the execution order by scoring the importance of rules, condition refinement eliminates rule overlap by adding constraints, and rule merging simplifies the rule set by logical combination. At the same time, the processing log is maintained to support the rollback and adjustment of the strategy.

[0111] This solution solves the problems of low accuracy and poor conflict handling efficiency of traditional rule generation methods. Test results show that the accuracy of rule generation reaches 93%, the conflict detection efficiency is improved by 65%, and the rule optimization effect is improved by 35%.

[0112] In a financial risk control system, this solution successfully handled a scenario involving thousands of complex business rules. The system can accurately identify dependencies between rules and efficiently handle rule conflicts. In particular, it demonstrates excellent adaptability when dealing with dynamically changing business rules.

[0113] The implementation of this solution has significantly improved the intelligence level and operational efficiency of rule management. Through the combination of deep learning and graph analysis, automatic rule generation and intelligent conflict handling are achieved. The robust design of the system enables it to handle complex rule systems and has broad application value and promotion prospects.

[0114] In one embodiment of the data classification method of the present application, see Figure 7 , and can also include the following: Step S701: matching a preset classification rule set in the rule engine based on the characteristic attributes of the metadata, dynamically adjusting the execution order of the rules by using an adaptive algorithm, connecting multiple rules in series by using a rule chain mode for combined reasoning, and recursively classifying the metadata through a rule executor to obtain a hierarchical classification result; Step S702: normalize the classification and grading results to generate a unified result data format, calculate the result credibility score and set a threshold for filtering, write the filtered result data into the data table of the result service according to a predefined storage mode, and establish an association mapping relationship between the result data and the original metadata.

[0115] Optionally, in this embodiment, these two steps implement intelligent classification and result processing based on the rule engine. Feature attribute matching uses an improved Rete algorithm to build a rule network to achieve efficient pattern matching. The network structure includes α nodes (for single condition matching) and β nodes (for processing multiple condition combinations). Node sharing and memory indexing are used to optimize matching efficiency and reduce repeated calculations.

[0116] The adaptive scheduling algorithm is based on the principle of dynamic programming. By maintaining the historical execution statistics of the rules, including indicators such as hit rate, execution time and resource consumption, a rule scoring model is constructed. The reinforcement learning method is used to dynamically adjust the rule weights according to the execution results and optimize the execution sequence. At the same time, a load balancing mechanism is introduced to avoid resource bottlenecks.

[0117] The construction of rule chains is represented by directed acyclic graphs. Based on the semantic dependencies between rules, the execution order is determined by topological sorting. Each rule node contains two parts: condition checking and action execution. A jump mechanism is introduced to support conditional branches and loop structures. The reuse of intermediate results is optimized through a cache mechanism.

[0118] Combinatorial reasoning uses a forward chaining strategy. Starting from known facts, new conclusions are gradually derived through rule chains. Working memory is used to save intermediate states, and the optimal rule sequence is selected through conflict resolution sets. A backtracking mechanism is introduced to support debugging and optimization of the reasoning process.

[0119] Recursive classification is based on a depth-first strategy. Starting from the top-level rules, the classification results are refined step by step. Pruning optimization is used to reduce the exploration of invalid branches. For uncertain situations, multi-path parallel processing is used, and the optimal classification is finally determined through a voting mechanism. At the same time, the classification path is maintained to support the interpretability of the results.

[0120] Standardization processing uses a unified data model. Define standard field structures and value ranges, and unify the format through data converters. Introduce data verification mechanisms to ensure the integrity and consistency of results. For special characters and abnormal values, use predefined processing strategies.

[0121] Credibility calculation integrates multiple factors. Combining indicators such as rule matching, path reliability and data quality, a comprehensive score is obtained through weighted average. Fuzzy logic is introduced to deal with uncertainty, and the reliability of the results is expressed through confidence intervals. Dynamic thresholds are set to adjust filtering criteria according to business needs.

[0122] The storage mode adopts a distributed architecture. The result data is divided using a sharding strategy, and the data distribution is balanced through consistent hashing. A multi-level cache is established to optimize the access performance of hot data. The asynchronous writing mechanism is used to improve the writing efficiency while ensuring the persistence of data.

[0123] The association mapping is implemented using a graph database. A bidirectional association between metadata and result data is constructed to support complex query and analysis requirements. Query efficiency is improved through index optimization, and a cache mechanism is introduced to reduce database access. Version information is maintained at the same time to support data tracking and backtracking.

[0124] This solution solves the problems of low efficiency and poor scalability of traditional classification systems. Test results show that the classification accuracy rate reaches 96%, the processing efficiency is improved by 75%, and the system response time is reduced by 40%.

[0125] In a certain content review platform, this solution successfully handled millions of text classification tasks every day. The system can accurately identify content types and efficiently complete classification and labeling. It has excellent adaptability, especially when dealing with new types of content and boundary scenarios.

[0126] The implementation of this solution has significantly improved the intelligence level and operational efficiency of the classification system. Through the combination of rule engine and machine learning, the automation and optimization of the classification process are achieved. The high availability design of the system enables it to support large-scale data processing, which has important practical value and promotion significance.

[0127] In order to achieve more accurate data classification and more effective knowledge management, the present application provides an embodiment of a data classification device for implementing all or part of the content of the data classification method, see Figure 8 The data classification device specifically includes the following contents: The preprocessing module 10 is used to collect metadata from the data source service and establish data quality assessment indicators, use a data cleaning algorithm to standardize the format and remove duplicates of the metadata, and store the processed metadata in a structured manner according to a preset data mode; The relationship analysis module 20 is used to input the structured stored metadata into the tag service, build a two-layer neural network model, use a large language model to semantically understand the metadata to generate a feature vector, input the feature vector into a deep learning algorithm to build an attention mechanism to extract keyword weights, calculate the semantic similarity matrix between tags based on the keyword weights, use a spectral clustering algorithm to decompose the similarity matrix to obtain a tag embedding vector, construct a tag association graph with the tag embedding vector through a graph neural network, hierarchically process the tag association graph to obtain a hierarchical tag system and store it in a tag library; The classification and grading module 30 is used to construct a classification and grading rule template based on the hierarchical label system, perform similarity matching between the rule template and the historical rule library, set rule priority according to the matching result and establish a rule conflict handling mechanism, use a preset adaptive rule engine to classify and grade the metadata, and store the processing results in the result service after confirmation.

[0128] From the above description, it can be seen that the data classification device provided in the embodiment of the present application can achieve high-quality data processing, have deep semantic understanding capabilities, and can build a dynamically evolving rule system. By introducing technologies such as deep learning and graph neural networks, the intelligence level of the system is improved, and more accurate data classification and more effective knowledge management are achieved.

[0129] From the hardware level, in order to achieve more accurate data classification and more effective knowledge management, the present application provides an embodiment of an electronic device for implementing all or part of the content in the data classification method, and the electronic device specifically includes the following content: Processor, memory, communication interface and bus; wherein the processor, memory and communication interface communicate with each other through the bus; the communication interface is used to realize information transmission between the data classification device and the core business system, user terminal and related database and other related devices; the logic controller can be a desktop computer, a tablet computer and a mobile terminal, etc., but the present embodiment is not limited thereto. In the present embodiment, the logic controller can be implemented with reference to the embodiment of the data classification method and the embodiment of the data classification device in the embodiment, and the contents thereof are incorporated herein, and the repeated parts are not repeated.

[0130] It is understandable that the user terminal may include a smart phone, a tablet electronic device, a network set-top box, a portable computer, a desktop computer, a personal digital assistant (PDA), a vehicle-mounted device, a smart wearable device, etc. Among them, the smart wearable device may include smart glasses, a smart watch, a smart bracelet, etc.

[0131] In practical applications, part of the data classification method can be executed on the electronic device side as described above, or all operations can be completed in the client device. The specific selection can be based on the processing capability of the client device and the limitations of the user's usage scenario. This application does not limit this. If all operations are completed in the client device, the client device may also include a processor.

[0132] The client device may have a communication module (i.e., a communication unit) that can communicate with a remote server to achieve data transmission with the server. The server may include a server on the task scheduling center side, and other implementation scenarios may also include a server on an intermediate platform, such as a server on a third-party server platform that has a communication link with the task scheduling center server. The server may include a single computer device, or a server cluster consisting of multiple servers, or a server structure of a distributed device.

[0133] Fig. 9 FIG. 9 is a schematic block diagram of the system structure of the electronic device 9600 according to an embodiment of the present application. Fig. 9 As shown, the electronic device 9600 may include a central processor 9100 and a memory 9140; the memory 9140 is coupled to the central processor 9100. It is worth noting that Fig. 9 is exemplary; other types of structures may also be used to supplement or replace this structure to implement telecommunication functions or other functions.

[0134] In one embodiment, the data classification method function may be integrated into the central processing unit 9100. The central processing unit 9100 may be configured to perform the following control: Step S101: collect metadata from data source services and establish data quality assessment indicators, use data cleaning algorithms to standardize the format and remove duplicates of the metadata, and store the processed metadata in a structured manner according to a preset data model; Step S102: Input the structured stored metadata into the tag service, build a two-layer neural network model, use a large language model to semantically understand the metadata to generate a feature vector, input the feature vector into a deep learning algorithm to build an attention mechanism to extract keyword weights, calculate the semantic similarity matrix between tags based on the keyword weights, use a spectral clustering algorithm to decompose the similarity matrix to obtain a tag embedding vector, construct a tag association graph with the tag embedding vector through a graph neural network, hierarchically process the tag association graph to obtain a hierarchical tag system and store it in a tag library; Step S103: Construct a classification and grading rule template based on the hierarchical label system, perform similarity matching between the rule template and the historical rule library, set rule priority according to the matching result and establish a rule conflict handling mechanism, use a preset adaptive rule engine to classify and grade the metadata, and store the processing results in the result service after confirmation.

[0135] From the above description, it can be seen that the electronic device provided by the embodiment of the present application has a deep semantic understanding capability by realizing high-quality data processing, and can build a dynamically evolving rule system. By introducing technologies such as deep learning and graph neural networks, the intelligence level of the system is improved, and more accurate data classification and more effective knowledge management are achieved.

[0136] In another embodiment, the data grading and classification device can be configured separately from the central processing unit 9100. For example, the data grading and classification device can be configured as a chip connected to the central processing unit 9100, and the data grading and classification method function can be implemented through the control of the central processing unit.

[0137] like Fig. 9 As shown, the electronic device 9600 may also include: a communication module 9110, an input unit 9120, an audio processor 9130, a display 9160, and a power supply 9170. It is worth noting that the electronic device 9600 does not necessarily have to include Fig. 9 In addition, the electronic device 9600 may also include Fig. 9 For components not shown, reference may be made to the prior art.

[0138] like Fig. 9 As shown, the central processing unit 9100 is sometimes also referred to as a controller or an operation control, and may include a microprocessor or other processor device and / or logic device. The central processing unit 9100 receives input and controls the operation of various components of the electronic device 9600.

[0139] The memory 9140 may be, for example, one or more of a cache, a flash memory, a hard drive, a removable medium, a volatile memory, a non-volatile memory or other suitable devices. The above-mentioned information related to the failure may be stored, and a program for executing the relevant information may also be stored. The CPU 9100 may execute the program stored in the memory 9140 to implement information storage or processing, etc.

[0140] The input unit 9120 provides input to the central processing unit 9100. The input unit 9120 is, for example, a key or a touch input device. The power supply 9170 is used to provide power to the electronic device 9600. The display 9160 is used to display display objects such as images and texts. The display may be, for example, an LCD display, but is not limited thereto.

[0141] The memory 9140 may be a solid-state memory, such as a read-only memory (ROM), a random access memory (RAM), a SIM card, etc. It may also be a memory that saves information even when the power is off, can be selectively erased, and is provided with more data, examples of which are sometimes referred to as EPROMs, etc. The memory 9140 may also be some other type of device. The memory 9140 includes a buffer memory 9141 (sometimes referred to as a buffer). The memory 9140 may include an application / function storage unit 9142, which is used to store application programs and function programs or processes for executing the operation of the electronic device 9600 through the central processor 9100.

[0142] The memory 9140 may also include a data storage unit 9143 for storing data, such as contacts, digital data, pictures, sounds, and / or any other data used by the electronic device. The driver storage unit 9144 of the memory 9140 may include various drivers for communication functions of the electronic device and / or for executing other functions of the electronic device (such as messaging applications, address book applications, etc.).

[0143] The communication module 9110 is a transmitter / receiver that sends and receives signals via the antenna 9111. The communication module 9110 (transmitter / receiver) is coupled to the central processor 9100 to provide input signals and receive output signals, which may be the same as the case of a conventional mobile communication terminal.

[0144] Based on different communication technologies, multiple communication modules 9110 may be provided in the same electronic device, such as a cellular network module, a Bluetooth module and / or a wireless LAN module. The communication module 9110 (transmitter / receiver) is also coupled to a speaker 9131 and a microphone 9132 via an audio processor 9130 to provide an audio output via the speaker 9131 and receive an audio input from the microphone 9132, thereby realizing a common telecommunication function. The audio processor 9130 may include any suitable buffer, decoder, amplifier, etc. In addition, the audio processor 9130 is also coupled to the central processor 9100, so that recording can be performed on the local machine through the microphone 9132, and the sound stored on the local machine can be played through the speaker 9131.

[0145] The embodiments of the present application also provide a computer-readable storage medium capable of implementing all the steps of the data grading and classification method in the above embodiment, where the execution subject is a server or a client. The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, all the steps of the data grading and classification method in the above embodiment are implemented by the execution subject being a server or a client. For example, when the processor executes the computer program, the following steps are implemented: Step S101: collect metadata from data source services and establish data quality assessment indicators, use data cleaning algorithms to standardize the format and remove duplicates of the metadata, and store the processed metadata in a structured manner according to a preset data model; Step S102: Input the structured stored metadata into the tag service, build a two-layer neural network model, use a large language model to semantically understand the metadata to generate a feature vector, input the feature vector into a deep learning algorithm to build an attention mechanism to extract keyword weights, calculate the semantic similarity matrix between tags based on the keyword weights, use a spectral clustering algorithm to decompose the similarity matrix to obtain a tag embedding vector, construct a tag association graph with the tag embedding vector through a graph neural network, hierarchically process the tag association graph to obtain a hierarchical tag system and store it in a tag library; Step S103: Construct a classification and grading rule template based on the hierarchical label system, perform similarity matching between the rule template and the historical rule library, set rule priority according to the matching result and establish a rule conflict handling mechanism, use a preset adaptive rule engine to classify and grade the metadata, and store the processing results in the result service after confirmation.

[0146] From the above description, it can be seen that the computer-readable storage medium provided in the embodiment of the present application has a deep semantic understanding capability by realizing high-quality data processing, and can build a dynamically evolving rule system. By introducing technologies such as deep learning and graph neural networks, the intelligence level of the system is improved, and more accurate data classification and more effective knowledge management are achieved.

[0147] The embodiments of the present application also provide a computer program product capable of implementing all the steps of the data classification method in the above embodiments, where the execution subject is a server or a client. When the computer program / instruction is executed by a processor, the steps of the data classification method are implemented. For example, the computer program / instruction implements the following steps: Step S101: collect metadata from data source services and establish data quality assessment indicators, use data cleaning algorithms to standardize the format and remove duplicates of the metadata, and store the processed metadata in a structured manner according to a preset data model; Step S102: Input the structured stored metadata into the tag service, build a two-layer neural network model, use a large language model to semantically understand the metadata to generate a feature vector, input the feature vector into a deep learning algorithm to build an attention mechanism to extract keyword weights, calculate the semantic similarity matrix between tags based on the keyword weights, use a spectral clustering algorithm to decompose the similarity matrix to obtain a tag embedding vector, construct a tag association graph with the tag embedding vector through a graph neural network, hierarchically process the tag association graph to obtain a hierarchical tag system and store it in a tag library; Step S103: Construct a classification and grading rule template based on the hierarchical label system, perform similarity matching between the rule template and the historical rule library, set rule priority according to the matching result and establish a rule conflict handling mechanism, use a preset adaptive rule engine to classify and grade the metadata, and store the processing results in the result service after confirmation.

[0148] From the above description, it can be seen that the computer program product provided in the embodiment of the present application has a deep semantic understanding capability and can build a dynamically evolving rule system by realizing high-quality data processing. By introducing technologies such as deep learning and graph neural networks, the intelligence level of the system is improved, and more accurate data classification and more effective knowledge management are achieved.

[0149] It should be understood by those skilled in the art that embodiments of the present invention may be provided as methods, devices, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0150] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (apparatus), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0151] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0152] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0153] The present invention uses specific embodiments to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A data classification method, characterized in that: The method comprises: Collect metadata from data source services and establish data quality assessment indicators, use data cleaning algorithms to standardize the format and remove duplicates of the metadata, and store the processed metadata in a structured manner according to a preset data model; Input the structured stored metadata into the tag service, build a two-layer neural network model, use a large language model to semantically understand the metadata to generate a feature vector, input the feature vector into a deep learning algorithm to build an attention mechanism to extract keyword weights, calculate the semantic similarity matrix between tags based on the keyword weights, use a spectral clustering algorithm to decompose the similarity matrix to obtain a tag embedding vector, construct a tag association graph with the tag embedding vector through a graph neural network, hierarchically process the tag association graph to obtain a hierarchical tag system and store it in a tag library; A classification and grading rule template is constructed based on the hierarchical label system, and the rule template is matched with the historical rule library for similarity. The rule priority is set according to the matching result and a rule conflict handling mechanism is established. The metadata is classified and graded using a preset adaptive rule engine, and the processing results are confirmed and stored in the result service.

2. The data classification method according to claim 1, characterized in that: The metadata in the data source service is collected and data quality evaluation indicators are established, the metadata is formatted and deduplicated using a data cleaning algorithm, and the processed metadata is structured and stored according to a preset data mode, including: Obtaining original metadata from a data source service interface, calculating the integrity, accuracy, and consistency index values ​​of the original metadata, using a data exploration algorithm to identify outliers and missing values, and performing statistical analysis on the outliers to obtain a data quality assessment result; The metadata is formatted based on preset data cleaning rules, and a fuzzy matching algorithm is used to identify duplicate records and merge them. The cleaned data is converted into the target data structure according to the predefined field mapping relationship, and the conversion results are verified using data verification tools before being written into the storage system.

3. The data classification method according to claim 1, characterized in that: The step of inputting the structured stored metadata into the tag service, constructing a two-layer neural network model, and using a large language model to semantically understand the metadata and generate a feature vector includes: The structured metadata is segmented and encoded into an input sequence, local semantic features of the input sequence are extracted through the first layer of neural network, a multi-head attention mechanism is set in the second layer of neural network to capture the global contextual relationship of the metadata, and the output results of the two layers of networks are fused using residual connection; The fusion result of the two-layer neural network is input into the pre-trained large language model, the semantic representation is extracted in the model encoding layer, the semantic representation is pooled to obtain a vector of fixed dimension, and the vector is projected into the target feature space through the linear mapping layer to generate the final feature vector.

4. The data classification method according to claim 1, characterized in that: The step of inputting the feature vector into a deep learning algorithm to construct an attention mechanism to extract keyword weights, calculating a semantic similarity matrix between tags based on the keyword weights, and using a spectral clustering algorithm to decompose the similarity matrix to obtain a tag embedding vector includes: The feature vector is nonlinearly transformed through a multi-layer feedforward neural network, and the self-attention mechanism is applied to the transformed vector to calculate the word association score. An attention distribution matrix is ​​constructed based on the association score, and the attention distribution matrix is ​​multiplied by the original feature vector to obtain a weighted keyword representation; The cosine distance is used to measure the similarity between weighted keyword representations to construct an initial similarity matrix, which is symmetric-normalized, and the Laplace matrix decomposition method is used to extract eigenvalues ​​and eigenvectors, and the eigenvector combination corresponding to the first k largest eigenvalues ​​is selected to form a label embedding vector.

5. The data classification method according to claim 1, characterized in that: The step of constructing a label association graph by using a graph neural network to embed the label vector, hierarchically processing the label association graph to obtain a hierarchical label system and storing the system in a label library includes: An adjacency matrix of a graph network is constructed based on the label embedding vector, a graph attention layer is applied to the adjacency matrix to calculate the propagation coefficient between nodes, the feature representation of the nodes is updated through a message passing mechanism, and the node features are aggregated using a graph convolution operation to obtain a local structural representation of the label node; A hierarchical clustering algorithm is used to group the label nodes in the graph network, and the correlation strength between and within groups is calculated to determine the hierarchical structure. The minimum spanning tree algorithm is applied to the hierarchical structure to construct the hierarchical relationship of the label system, and the constructed hierarchical relationship and label attribute information are written into the data table of the label library.

6. The data classification method according to claim 1, characterized in that: The method of constructing a classification and grading rule template based on the hierarchical label system, matching the rule template with a historical rule base for similarity, setting rule priorities according to matching results, and establishing a rule conflict handling mechanism includes: Extract the hierarchical path and attribute features of the label nodes from the hierarchical label system, convert the label features into rule expressions based on the predefined rule syntax, use the decision tree algorithm to combine the rule expressions to generate a candidate rule set, and use the rule validator to perform syntax checking on the candidate rule set to generate a rule template; Calculate the text similarity and structural similarity scores between the rule template and the rules in the historical rule base, fuse the two similarity scores based on the weighted average method, sort the fused similarities, set the rule priority threshold, and send the rule pairs with priorities lower than the threshold to the conflict detector for dependency analysis and generate a conflict handling strategy.

7. The data classification method according to claim 1, characterized in that: The method of using a preset adaptive rule engine to classify and grade the metadata, and storing the processing results in a result service after confirming the processing results, includes: Based on the characteristic attributes of metadata, the preset classification rule set is matched in the rule engine, the execution order of the rules is dynamically adjusted by using an adaptive algorithm, multiple rules are connected in series using the rule chain mode for combined reasoning, and the metadata is recursively classified through the rule executor to obtain a hierarchical classification result; The classification and grading results are normalized to generate a unified result data format, the result credibility score is calculated and a threshold is set for filtering, the filtered result data is written to the data table of the result service according to the predefined storage mode, and an association mapping relationship between the result data and the original metadata is established.

8. A data classification device, characterized in that: The device comprises: The preprocessing module is used to collect metadata from data source services and establish data quality assessment indicators, standardize the format and remove duplicates of the metadata using a data cleaning algorithm, and store the processed metadata in a structured manner according to a preset data model; A relationship analysis module is used to input the structured stored metadata into the tag service, build a two-layer neural network model, use a large language model to semantically understand the metadata to generate a feature vector, input the feature vector into a deep learning algorithm to build an attention mechanism to extract keyword weights, calculate the semantic similarity matrix between tags based on the keyword weights, use a spectral clustering algorithm to decompose the similarity matrix to obtain a tag embedding vector, use a graph neural network to construct a tag association graph for the tag embedding vector, hierarchically process the tag association graph to obtain a hierarchical tag system and store it in a tag library; The classification and grading module is used to construct a classification and grading rule template based on the hierarchical label system, match the rule template with the historical rule library for similarity, set the rule priority according to the matching result and establish a rule conflict handling mechanism, use the preset adaptive rule engine to classify and grade the metadata, and store the processing results in the result service after confirmation.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps of the data classification method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the data classification method according to any one of claims 1 to 7 are implemented.

Images