Automatic data classification and grading system and method

Abstract

The invention discloses an automatic data classification and grading system and method, and relates to the field of intelligent grading, and the method comprises the steps: carrying out the summarization description of data assets to be processed through employing a data analysis and coding mode based on a large model and artificial intelligence, carrying out the semantic coding of the summarized abstract description of the data assets to be processed, and carrying out the semantic coding of the summarized abstract description of the data assets to be processed; meanwhile, semantic embedding coding is performed on each classification standard, so that each matching probability is intelligently generated according to principal component semantic matching representation between the processed data asset abstract description semantic feature and each classification standard semantic embedding feature; and taking the maximum corresponding classification standard as the category label of the to-be-processed data assets. According to the method, general description and semantic processing can be carried out on the data assets to intelligently create the classification labels, and it is ensured that the data can respond to changes of business requirements in time.

Description

Technical Field

[0001] The present application relates to the field of intelligent classification, and more specifically, to a data automatic classification and grading system and method. Background Art

[0002] In today's digital age, all industries have accumulated massive amounts of data assets, from the massive customer transaction records and credit information stored by financial institutions, to the user behavior logs and content creation data owned by Internet companies, to the product design drawings and production process data of manufacturing companies, etc. Data has become a key factor in promoting corporate development and enhancing competitiveness. However, the disorderly accumulation of these data assets has brought severe challenges to enterprises. Therefore, it is particularly important to effectively classify these data to meet the needs of efficient management and utilization.

[0003] Traditional data classification methods mainly rely on manual work and keyword matching, which is increasingly inefficient and error-prone. On the one hand, data management personnel within the enterprise need to spend a lot of time and energy to read through complex and diverse data documents and try to classify data based on vague empirical rules. This not only leads to slow progress in classification work, but also makes it difficult to meet the needs of rapid business iteration for timely classification and utilization of data. On the other hand, simple keyword matching or statistical methods cannot deeply understand the content and background of the data, ignoring the deep semantic information contained therein, resulting in inaccurate classification results.

[0004] Therefore, an automated data classification and grading solution is desired that can quickly and accurately identify the key features of data assets so as to achieve scientific classification and grading of data assets. Summary of the invention

[0005] In order to solve the above technical problems, this application is proposed.

[0006] According to one aspect of the present application, a method for automatic data classification and grading is provided, which includes: obtaining data assets to be processed; extracting a set of classification standards from a classification standard library; performing a data asset summary description on the data assets to be processed to obtain a summary description of the data assets to be processed; performing semantic encoding on the summary description of the data assets to be processed to obtain a semantic encoding vector of the summary description of the data assets to be processed; performing semantic embedding encoding on each classification standard in the set of classification standards to obtain a set of classification standard semantic embedding encoding vectors; performing data asset semantic principal component matching compensation on each classification standard semantic embedding encoding vector in the set of the summary description semantic encoding vector of the data assets to be processed and the classification standard semantic embedding encoding vector to obtain a set of data asset-classification standard semantic matching encoding vectors; obtaining a set of matching probabilities based on the set of data asset-classification standard semantic matching encoding vectors; and determining the category label of the data asset to be processed based on the classification standard corresponding to the largest one in the set of matching probabilities.

[0007] According to another aspect of the present application, a data automatic classification and grading system is provided, which includes: a data asset acquisition module for acquiring data assets to be processed; a classification standard extraction module for extracting a collection of classification standards from a classification standard library; a data asset summary description module for performing a data asset summary description on the data assets to be processed to obtain a summary description of the data assets to be processed; a summary description semantic encoding module for performing semantic encoding on the summary description of the data assets to be processed to obtain a semantic encoding vector of the summary description of the data assets to be processed; and a classification standard semantic embedding module for performing semantic embedding encoding on each classification standard in the collection of classification standards. The invention relates to a method for processing a data asset summary description semantic coding vector to obtain a set of classification standard semantic embedding coding vectors; a semantic principal component matching module, which is used to perform data asset semantic principal component matching compensation on the semantic coding vector of the summary description of the data asset to be processed and each classification standard semantic embedding coding vector in the set of the classification standard semantic embedding coding vector to obtain a set of data asset-classification standard semantic matching coding vectors; a matching probability generating module, which is used to obtain a set of matching probabilities based on the set of data asset-classification standard semantic matching coding vectors; and a category label determining module, which is used to determine the category label of the data asset to be processed based on the classification standard corresponding to the largest one in the set of matching probabilities.

[0008] Compared with the prior art, the present application provides a data automatic classification and grading system and method, which obtains the data assets to be processed, extracts a set of classification standards from the classification standard library, and uses data analysis and coding methods based on large models and artificial intelligence to summarize the data assets to be processed, and then semantically encodes the summarized summary description of the data assets to be processed, and at the same time, semantically embeds each classification standard, so as to intelligently generate each matching probability based on the principal component semantic matching representation between the semantic features of the summary description of the processed data assets and the semantic embedding features of each classification standard, and uses the classification standard with the largest corresponding one as the category label of the data asset to be processed. The present application can summarize and semantically process data assets, extract their inherent deep meaning to intelligently create classification labels, thereby greatly reducing manual intervention, significantly improving classification speed, and ensuring that data can respond to changes in business needs in a timely manner. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] By describing the embodiments of the present application in more detail in conjunction with the accompanying drawings, the above and other purposes, features and advantages of the present application will become more apparent. The accompanying drawings are used to provide a further understanding of the embodiments of the present application and constitute a part of the specification. Together with the embodiments of the present application, they are used to explain the present application and do not constitute a limitation of the present application. In the accompanying drawings, the same reference numerals generally represent the same components or steps.

[0010] Figure 1 The present invention is a flowchart of a method for automatic classification and grading of data according to an embodiment of the present application.

[0011] Figure 2 Flow chart of step S160 in the method for automatic data classification and grading according to an embodiment of the present application.

[0012] Figure 3 This is a flowchart of step S163 in the method for automatic data classification and grading according to an embodiment of the present application.

[0013] Figure 4 It is a block diagram of a data automatic classification and grading system according to an embodiment of the present application. DETAILED DESCRIPTION

[0014] The embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments described herein, but rather these embodiments are provided for a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0015] It should be understood that the various steps described in the method embodiments of the present disclosure may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this respect.

[0016] It is worth noting that in this application, all actions to obtain signals, information or data are carried out in compliance with the relevant data protection laws and policies of the country where they are located and with the authorization given by the owner of the corresponding device.

[0017] In today's digital age, all industries have accumulated massive amounts of data assets, from the massive customer transaction records and credit information stored by financial institutions, to the user behavior logs and content creation data owned by Internet companies, to the product design drawings and production process data of manufacturing companies, etc. Data has become a key factor in promoting corporate development and enhancing competitiveness. However, the disorderly accumulation of these data assets has brought severe challenges to enterprises. Therefore, it is particularly important to effectively classify these data to meet the needs of efficient management and utilization.

[0018] Traditional data classification methods mainly rely on manual work and keyword matching, which is increasingly inefficient and error-prone. On the one hand, data management personnel within the enterprise need to spend a lot of time and energy to read through complex and diverse data documents and try to classify data based on vague empirical rules. This not only leads to slow progress in classification work, but also makes it difficult to meet the needs of rapid business iteration for timely classification and utilization of data. On the other hand, simple keyword matching or statistical methods cannot deeply understand the content and background of the data, ignoring the deep semantic information contained therein, resulting in inaccurate classification results.

[0019] Therefore, in response to the above technical problems, the technical concept of the present application is to obtain the data assets to be processed, and extract a set of classification standards from the classification standard library, and use data analysis and coding methods based on large models and artificial intelligence to summarize the data assets to be processed, and then semantically encode the summarized summary description of the data assets to be processed, and at the same time, semantically embed code each classification standard, so as to intelligently generate each matching probability based on the principal component semantic matching representation between the semantic features of the summary description of the processed data assets and the semantic embedding features of each classification standard, and use the classification standard with the largest corresponding one as the category label of the data asset to be processed. The present application can summarize and semantically process data assets, extract their inherent deep meaning to intelligently create classification labels, thereby greatly reducing manual intervention, significantly improving the classification speed, and ensuring that data can respond to changes in business needs in a timely manner.

[0020] Figure 1Flow chart of the data automatic classification and grading method according to the embodiment of the present application. Figure 1 As shown, according to the embodiment of the present application, the method for automatic classification and grading of data includes: S110, obtaining data assets to be processed; S120, extracting a set of classification standards from a classification standard library; S130, performing a data asset summary description on the data assets to be processed to obtain a summary description of the data assets to be processed; S140, performing semantic encoding on the summary description of the data assets to be processed to obtain a semantic encoding vector of the summary description of the data assets to be processed; S150, performing semantic embedding encoding on each classification standard in the set of classification standards to obtain a set of classification standard semantic embedding encoding vectors; S160, performing data asset semantic principal component matching compensation on each classification standard semantic embedding encoding vector in the set of the semantic encoding vector of the summary description of the data assets to be processed and the classification standard semantic embedding encoding vector to obtain a set of data asset-classification standard semantic matching encoding vectors; S170, obtaining a set of matching probabilities based on the set of data asset-classification standard semantic matching encoding vectors; S180, determining the category label of the data asset to be processed based on the classification standard corresponding to the largest one in the set of matching probabilities.

[0021] In step S110, the data assets to be processed are obtained. It should be understood that the data assets to be processed refer to various data sets owned by enterprises, organizations or institutions that need to be classified, managed and subsequently utilized. For example, historical records generated by interactions between customers and enterprises, production order data of financial statement data production enterprises, etc. Therefore, in order to sort out and classify these data assets to be processed according to established classification standards (such as according to characteristics such as nature, purpose, source, etc.), in the technical solution of this application, it is necessary to obtain the data assets to be processed. In this way, corresponding utilization methods can be adopted for different categories of data in the future. For example, for customer information data, it can be used to build customer portraits and formulate personalized marketing strategies; for financial information data, it can assist financial analysis, budget preparation and risk warning, etc. Only by obtaining data assets can their respective values ​​be accurately excavated and precise utilization be achieved.

[0022] Various business systems within an enterprise are the main data source channels. For example, the core transaction system, customer relationship management system (CRM), and credit management system of financial institutions will generate a large amount of customer transaction data and credit information; the website platform and mobile application background of Internet companies will record user behavior logs and content creation data; the product R&D and design software, enterprise resource planning (ERP) system, and production execution system (MES) of manufacturing companies are the source of product design drawings and production process data. In addition, enterprises may also obtain some data from the outside, such as market research data, industry report data, etc., which also need to be included in the category of data assets to be processed.

[0023] After determining the source of data, a key issue is data collection. For data collection in internal enterprise systems, ETL (Extract, Transform, Load) tools or data synchronization technology are usually used. ETL tools can extract data from the source database, clean, transform and integrate it according to predetermined rules, and then load it into the target storage location, such as a data warehouse or data lake. For example, in financial institutions, ETL tools can be used to extract transaction record data from the transaction system, perform format conversion, denoising and other operations on the data to ensure the quality and consistency of the data for subsequent analysis and processing. For user behavior data collection of Internet companies, log collection tools such as Fluentd and Logstash may be used. These tools can efficiently collect user behavior logs distributed on different servers and store them centrally in a designated location.

[0024] In the process of data collection, data quality control is crucial. Since data may come from different systems and channels, there are problems such as inconsistent data formats, missing data, and data errors. Therefore, it is necessary to detect and repair data quality during or after the collection process. Data quality tools such as Informatica Data Quality and IBM InfoSphere Information Analyzer can be used to perform operations such as integrity checks, accuracy verification, and consistency checks on data. For example, in the production process data collection of manufacturing companies, it may be found that data in certain production links is missing. At this time, the missing data can be repaired by using data interpolation algorithms or by associating and supplementing with other related data to ensure data integrity and availability.

[0025] Data storage is also an important part of obtaining data assets to be processed. According to the characteristics of the data and the needs of subsequent processing, choose the appropriate storage technology and architecture. For structured data, such as transaction records of financial institutions and production order data of manufacturing enterprises, relational databases (such as MySQL, Oracle, SQLServer, etc.) are commonly used storage methods, which have the advantages of strong transaction processing capabilities, clear data structure, easy query and management. For semi-structured and unstructured data, such as user behavior logs and content creation data of Internet companies, product design drawings of manufacturing companies, data lakes (such as HDFS or cloud storage platforms based on Hadoop) or NoSQL databases (such as MongoDB, Cassandra, etc.) are more suitable. Data lakes can store a large amount of different types of data in their original format, which facilitates subsequent exploratory analysis and data mining; NoSQL databases have advantages in processing large-scale, high-concurrency, and flexible data structures. When storing data, data security and privacy protection must also be considered. Enterprises need to comply with relevant data protection regulations and encrypt and store sensitive data and control access. For example, sensitive data such as customer credit information of financial institutions needs to be encrypted using encryption algorithms (such as AES, RSA, etc.) during storage to ensure the security of data on the storage medium. At the same time, by setting strict user permissions and access policies, only authorized personnel can access the corresponding data to prevent data leakage and abuse.

[0026] In addition, the availability and usability of data are also aspects that need attention. In order to facilitate subsequent data processing and analysis, the stored data needs to have a good organization and index structure. For relational databases, the query efficiency of data can be improved by creating appropriate table structures, indexes, and views; for data lakes and NoSQL databases, metadata management tools and data catalog technologies can be used to classify, label, and index the stored data so that data assets can be found and accessed quickly and accurately. For example, in the data lake of an Internet company, the Hive metadata management tool can be used to classify and index user behavior log data so that data analysts can quickly locate and extract the required data for analysis.

[0027] In the process of acquiring data assets to be processed, real-time data processing may also be involved. For some time-sensitive business scenarios, such as real-time risk monitoring of financial transactions and real-time user recommendations of Internet companies, real-time data processing technologies such as stream computing frameworks (such as Apache Flink, Apache Storm, etc.) need to be adopted. These stream computing frameworks can process and analyze data in real time as it is generated, ensuring the timeliness and effectiveness of the data.

[0028] In step S120, a set of classification standards is extracted from the classification standard library. Accordingly, the classification standard is a series of rules, attributes or characteristic dimensions used to distinguish different categories of data assets. It is a criterion formulated based on factors such as data management objectives, business needs and related systems. However, data assets themselves are diverse and complex, and a single classification standard is often not enough to accurately and comprehensively classify all data. Therefore, in order to comprehensively consider the characteristics of data in multiple dimensions such as nature, purpose, and source, this application extracts a set of classification standards from the classification standard library, so as to classify the data to be processed in a more detailed and standardized manner by adopting a unified standard form, thereby improving the effectiveness of data management and utilization.

[0029] The establishment of a classification standard library is the foundation. The construction of this library is not achieved overnight. It requires extensive collection and organization of information from different channels. The business rules and processes within the enterprise are one of the important sources. For example, in manufacturing enterprises, different production processes, different types and specifications of products can all serve as the basis for classification standards. From raw material procurement to finished product delivery, each stage has its own specific business logic, which is sorted out and refined before being incorporated into the classification standard library.

[0030] At the same time, industry norms and standards also play a key role in the construction of the classification standard library. Taking the medical industry as an example, the International Classification of Diseases (ICD) standards and the classification specifications of medical devices are all guidelines that must be followed. These standards ensure that companies are consistent and compatible with the industry as a whole when processing medical data, allowing data to be effectively communicated and shared between different institutions.

[0031] When the classification standard library is initially built, how to efficiently extract the set of classification standards from it becomes the key. Usually, this process is achieved with the help of a database management system. If a relational database such as MySQL or Oracle is used, the classification standards will be stored in a structured form in a data table. Each classification standard may contain fields such as the standard name, standard description, scope of application, and industry to which it belongs. For example, in the classification standard table of the financial industry, a record may be "Credit risk classification standard, used to assess customer credit default risk, applicable to all credit businesses, financial industry". By writing SQL query statements, you can extract a set of classification standards related to the financial industry and credit.

[0032] In order to improve extraction efficiency, indexing technology is essential. In the database, create indexes for fields that are frequently used for retrieval, such as standard names or applicable scope fields. In this way, when performing extraction operations, the database system can quickly locate records that meet the conditions, greatly reducing query time. For example, in an Internet enterprise database with a large number of classification standards, after creating an index for the "content_type" field, when extracting classification standards related to specific content types (such as videos, pictures, etc.), the system can respond quickly, improving the efficiency of the entire data classification and grading process.

[0033] In addition to database technology, metadata management also plays an important role in the extraction process. Metadata records detailed information about the classification standard, such as the creation time, modification history, source, etc. Through the metadata management system, when extracting the classification standard collection, you can filter and filter according to the metadata. For example, only extract the most recently updated standards, or standards from a specific source, to ensure that the extracted collection meets the current business needs and the actual situation of data processing.

[0034] In the actual operation of an enterprise, user rights management is also a factor that needs to be considered when extracting a set of classification standards. Users in different departments or positions may only need to access and use specific classification standards. For example, the financial department of an enterprise may only need standards related to the classification of financial data, such as income classification standards, cost classification standards, etc.; while the marketing department is more concerned with customer data classification standards, such as customer segmentation standards, customer behavior classification standards, etc. Therefore, during the extraction process, the user rights management module will be combined to determine the scope of classification standards that the user can access based on the user's identity and role to ensure the security and confidentiality of the data.

[0035] In step S130, a data asset summary description is performed on the data asset to be processed to obtain a summary description of the data asset to be processed. Specifically, in an embodiment of the present application, a data asset summary description is performed on the data asset to be processed to obtain a summary description of the data asset to be processed, including: inputting the data asset to be processed into a data asset summary descriptor based on a large language model to obtain a summary description of the data asset to be processed. It should be understood that, considering that the data asset to be processed is often of high complexity, the amount of data is usually very large, which contains key semantic information about the data asset, and these semantic information are crucial for accurate classification and grading, but there are also some irrelevant semantics accordingly. Therefore, in order to be able to extract key information from complex data, the present application inputs the data asset to be processed into a data asset summary descriptor based on a large language model to use the large language model to deeply understand the text content, capture contextual relationships and extract key information therefrom, so as to obtain a more accurate and meaningful summary description of the data asset to be processed. It should be understood that the large language model can deeply understand the semantics of natural language after being trained with a large amount of text data. It can mine deep semantic associations and key content based on the text part of the data assets, and then form a valuable summary description, which can more accurately grasp the connotation of the data than the traditional method based on simple rules or keyword matching. In particular, in a specific embodiment of the present application, the large language model is gpt-4.

[0036] In step S140, the summary description of the data asset to be processed is semantically encoded to obtain a semantic encoding vector of the summary description of the data asset to be processed. Specifically, in an embodiment of the present application, the summary description of the data asset to be processed is semantically encoded to obtain a semantic encoding vector of the summary description of the data asset to be processed, including: segmenting the summary description of the data asset to be processed and inputting it into a semantic encoder containing a word embedding layer to obtain the semantic encoding vector of the summary description of the data asset to be processed. Accordingly, considering that the summary description of the data asset to be processed has simplified the original data, in order to have a deeper understanding of its meaning and to capture and mine out deeper intrinsic semantic meanings, in the technical solution of the present application, the summary description of the data asset to be processed is semantically encoded to map the text into a high-dimensional vector space, so that texts with similar meanings are closer in the vector space, thereby better capturing the semantic information of the text and obtaining a semantic encoding vector of the summary description of the data asset to be processed. In particular, in a specific embodiment of the present application, the summary of the data asset to be processed is first segmented to divide the description information of an entire text into multiple meaningful word units, and then each word is input into a semantic encoder including a word embedding layer to extract and encode the semantic information of the text based on the word level, capturing the semantic meaning contained in the context between each word to generate a semantic encoding vector describing the summary of the data asset to be processed that is rich in semantic information.

[0037] Specifically, the summary description of the data asset to be processed is segmented and input into a semantic encoder containing a word embedding layer to obtain a semantic encoding vector of the summary description of the data asset to be processed. The specific processing process is as follows: Assume that there is a summary description of a data asset to be processed as "The financial statements of Customer A show that its operating income in the third quarter has increased significantly, mainly from new business expansion."

[0038] First, we perform word segmentation and obtain words such as "customer", "A", "financial statements", "show", "its", "in", "third quarter", "operating income", "has", "significant growth", "mainly", "sourced from", "new business", and "expansion".

[0039] Next is the semantic encoder that contains the word embedding layer. The word embedding layer converts each word segment into a vector representation. For example, the word "customer" may be converted into a 200-dimensional vector, where the value of each dimension represents the position information of the word in a specific semantic space, which is obtained based on a large amount of text corpus training. Through the word embedding layer, all word segments have corresponding vector forms.

[0040] After the word embedding layer, there is a hidden layer. This hidden layer can be in the form of a fully connected layer, assuming it has 128 neurons. It will perform linear transformation and nonlinear activation on the input word vector. For example, for the word vector of "financial statement", after calculation in the hidden layer, it will integrate the semantic information of surrounding words such as "customer" and "operating income", and further enhance its semantic expression through activation functions (such as ReLU function). In this way, each word vector can capture richer contextual semantic associations after passing through the hidden layer.

[0041] In order to extract local semantic features, a convolution layer is added after the hidden layer. The convolution layer sets convolution kernels of different sizes, such as convolution kernels of size 3 and 4. For the convolution kernel of size 3, it will slide on the input feature vector sequence. When encountering the three consecutive word vectors of "third quarter", "operating income", and "have", it extracts the local semantic features of their combination, such as the feature that indicates the correlation between time and income. Convolution kernels of different sizes can obtain semantic information of multiple scales and enhance the ability to understand text.

[0042] A pooling layer is connected after the convolution layer, using the maximum pooling operation. The pooling layer performs dimensionality reduction on the feature map output by the convolution layer, and selects the maximum value in each feature map as a representative. For example, after the convolution and pooling operations, the important semantic features of the phrase "operating income has increased significantly" may be highlighted, while other relatively minor information is filtered out, thereby obtaining a more representative semantic feature representation.

[0043] Finally, after a series of processing, the feature vector is converted into the final semantic encoding vector of the summary description of the data asset to be processed through an output layer. The dimension of this semantic encoding vector may be 64 dimensions, which integrates the key semantic information of the entire input text and can be used for subsequent matching with classification standards.

[0044] In step S150, semantic embedding coding is performed on each classification standard in the set of classification standards to obtain a set of classification standard semantic embedding coding vectors. Specifically, in an embodiment of the present application, semantic embedding coding is performed on each classification standard in the set of classification standards to obtain a set of classification standard semantic embedding coding vectors, including: using a semantic encoder based on the Bert model to semantically embedding coding each classification standard in the set of classification standards to obtain a set of classification standard semantic embedding coding vectors. It should be understood that, considering that each classification standard usually contains terms and concepts in a specific field, therefore, in order to capture these deep semantic information, improve the comprehension of the classification standard, and facilitate subsequent comparison and analysis with the semantic data to be processed, in the technical solution of the present application, semantic embedding coding is performed on each classification standard in the set of classification standards to map each standard to the same semantic space, and obtain a set of classification standard semantic embedding coding vectors. In particular, in a specific embodiment of the present application, each classification standard in the set of classification standards is semantically embedded and encoded by using a semantic encoder based on the Bert model to map all classification standards into a common vector space, while being able to capture the deep semantic information in the classification standard text, thereby obtaining a set of classification standard semantic embedding coding vectors.

[0045] In step S160, the semantic principal component matching compensation of the data asset semantics is performed on the semantic coding vector of the summary description of the data asset to be processed and each classification standard semantic embedding coding vector in the set of the classification standard semantic embedding coding vector to obtain a set of data asset-classification standard semantic matching coding vectors. Furthermore, it is considered that there is a semantically critical and important deep correlation between the semantic coding vector of the summary description of the data asset to be processed and each classification standard semantic embedding coding vector. Moreover, different data asset summary descriptions and classification standards often have an imbalance in the expression of semantic features. It is possible that some key semantic features are not fully reflected in the coding vector, or some features are relatively weakened due to factors such as the characteristics of the data itself and the coding method. Therefore, in order to deeply analyze the semantic information contained in the two vectors, dig out the semantic connections hidden beneath the surface, focus on and reasonably compensate for key features, and thus more accurately measure the degree of matching between the two, the present application introduces a data asset semantic principal component matching compensation mechanism to match and encode the semantic coding vector of the summary description of the data asset to be processed and each classification standard semantic embedded coding vector in the set of the classification standard semantic embedded coding vector, and obtain a set of data asset-classification standard semantic matching coding vectors.

[0046] Figure 2Flowchart of step S160 in the data automatic classification and grading method according to an embodiment of the present application. Specifically, in the embodiment of the present application, Figure 2 As shown, the step S160 performs data asset semantic principal component matching compensation on each classification standard semantic embedded coding vector in the set of the data asset summary description semantic coding vector and the classification standard semantic embedded coding vector to obtain a set of data asset-classification standard semantic matching coding vectors, including: S161, performing feature principal component analysis on the data asset summary description semantic coding vector and the classification standard semantic embedded coding vector to obtain a set of data asset summary description semantic principal component coding vectors and a set of classification standard semantic embedded principal component coding vectors; S162, performing feature principal component analysis on the data asset summary description semantic coding vector and the classification standard semantic embedded principal component coding vector to obtain a set of data asset summary description semantic principal component coding vectors and a set of classification standard semantic embedded principal component coding vectors. The set of code vectors is reshaped to obtain a semantic principal component aggregation coding feature map of the summary description of the data asset to be processed and a classification standard semantic embedding principal component aggregation coding feature map; S163, calculating the data asset-classification standard semantic difference embedding compensation coding weight vector between the semantic principal component aggregation coding feature map of the summary description of the data asset to be processed and the classification standard semantic embedding principal component aggregation coding feature map; S164, based on the data asset-classification standard semantic difference embedding compensation coding weight vector, the set of the semantic principal component coding vectors of the summary description of the data asset to be processed and the set of the classification standard semantic embedding principal component coding vectors are globally averaged and aggregated to obtain the data asset-classification standard semantic matching coding vector.

[0047] Specifically, in step S161, feature principal component analysis is performed on the semantic coding vector of the summary description of the data asset to be processed and the semantic embedding coding vector of the classification standard to obtain a set of semantic principal component coding vectors of the summary description of the data asset to be processed and a set of semantic embedding principal component coding vectors of the classification standard. The formula of this process can be expressed as: ;in, is the semantic encoding vector describing the summary of the data asset to be processed, is the characteristic principal component analysis operation, It is through The calculated summary description sample covariance matrix of the data asset to be processed, is the principal component orthogonal matrix of the semantic features of the summary description of the data asset to be processed, For each data asset summary description semantic principal component encoding vector to be processed in the set of data asset summary description semantic principal component encoding vectors to be processed, A diagonal matrix of semantic features that summarizes the data assets to be processed, They are and The corresponding weight value, for The transposed matrix of is the taxonomy semantic embedding encoding vector, It is through The calculated classification standard sample covariance matrix, is the orthogonal matrix of the principal components of the classification standard semantic embedding features, is each classification standard semantic embedding principal component encoding vector in the set of classification standard semantic embedding principal component encoding vectors, is the classification standard semantic embedding feature diagonal matrix, They are and The corresponding weight value, for The transposed matrix of .

[0048] It should be understandable that in order to extract the key semantic principal components of the data asset summary description semantic features and the classification standard semantic embedding features, filter out relatively minor information, and enable subsequent processing to focus on the most representative and influential semantic features, feature principal component analysis is performed on the data asset summary description semantic encoding vector and the classification standard semantic embedding encoding vector to be processed to project the high-dimensional data into a lower-dimensional space, while retaining the maximum variance of the original data as much as possible, to obtain a set of data asset summary description semantic principal component encoding vectors and a set of classification standard semantic embedding principal component encoding vectors.

[0049] Specifically, in step S162, the set of semantic principal component coding vectors of the summary description of the data asset to be processed and the set of semantic embedding principal component coding vectors of the classification standard are reshaped to obtain the semantic principal component aggregate coding feature map of the summary description of the data asset to be processed and the classification standard semantic embedding principal component aggregate coding feature map. The formula of this process can be expressed as: ;in, For each data asset summary description semantic principal component encoding vector to be processed in the set of data asset summary description semantic principal component encoding vectors to be processed, is each classification standard semantic embedding principal component encoding vector in the set of classification standard semantic embedding principal component encoding vectors, For the reshape operation, It is the summary description of the data asset to be processed, the semantic principal component aggregation encoding feature map, It is the classification standard semantic embedding principal component aggregation encoding feature map.

[0050] Accordingly, each set after principal component analysis is reshaped to better reflect the relationship between different local semantic principal components and their layout in the overall semantic expression, and the semantic principal component aggregate coding feature map of the summary description of the data asset to be processed and the classification standard semantic embedding principal component aggregate coding feature map are obtained.

[0051] Figure 3 Flowchart of step S163 in the data automatic classification and grading method according to an embodiment of the present application. Specifically, in the embodiment of the present application, Figure 3 As shown, the step S163, calculating the data asset-classification standard semantic difference embedding compensation coding weight vector between the semantic principal component aggregation coding feature map of the summary description of the data asset to be processed and the classification standard semantic embedding principal component aggregation coding feature map, includes: S1631, performing semantic difference embedding on the semantic principal component aggregation coding feature map of the summary description of the data asset to be processed and the classification standard semantic embedding principal component aggregation coding feature map to obtain the semantic branch weight vector of the summary description of the data asset to be processed and the classification standard semantic embedding branch weight vector. The formula of this process can be expressed as: ;in, It is the summary description of the data asset to be processed, the semantic principal component aggregation encoding feature map, is the classification standard semantic embedding principal component aggregation encoding feature map, is the average pooling operation, and They are The corresponding first weight matrix and second weight matrix, and They are The corresponding first weight matrix and second weight matrix, is a nonlinear activation function, is the output activation function, is the semantic branch weight vector describing the summary of the data asset to be processed, is the classification standard semantic embedding branch weight vector.

[0052] S1632, calculating the data asset-classification standard semantic difference embedding compensation encoding weight vector between the to-be-processed data asset summary description semantic branch weight vector and the classification standard semantic embedding branch weight vector. The formula of this process can be expressed as: ;in, is the semantic branch weight vector describing the summary of the data asset to be processed, is the classification standard semantic embedding branch weight vector, It is subtracted by position point. To take the absolute value operation, is the data asset-taxonomy semantic difference embedding compensation encoding weight vector.

[0053] It should be understood that the semantic difference embedding is performed on the reshaped data features to be processed and the classification standard semantic features to quantify the semantic gap between the data assets to be processed and the classification standard and the importance of these differences, and the semantic branch weight vector of the summary description of the data assets to be processed and the semantic embedding branch weight vector of the classification standard are obtained. In this way, the contribution weight of each semantic principal component can be reasonably adjusted according to the difference size, so that the semantic matching is more in line with the actual semantic relationship.

[0054] Accordingly, considering that the semantic matching between data assets and classification standards is a two-way comparison process, it is necessary not only to consider the semantic difference between the data asset summary description and the classification standard, but also to comprehensively consider this two-way semantic difference to obtain a representation that comprehensively reflects the relationship between the two at the semantic weight level, and calculate the data asset-classification standard semantic difference embedding compensation encoding weight vector between the semantic branch weight vector of the data asset summary description to be processed and the classification standard semantic embedding branch weight vector.

[0055] Specifically, in the embodiment of the present application, the step S164, based on the data asset-classification standard semantic difference embedding compensation coding weight vector, performs global mean calculation and aggregation on the set of the semantic principal component coding vectors of the summary description of the data asset to be processed and the set of the semantic embedding principal component coding vectors of the classification standard to obtain the data asset-classification standard semantic matching coding vector, including: calculating the positional mean vector of the set of the semantic principal component coding vectors of the summary description of the data asset to be processed and the set of the semantic embedding principal component coding vectors of the classification standard to obtain the semantic principal component representation coding vector of the summary description of the data asset to be processed and the classification standard semantic embedding principal component representation coding vector. The formula of this process can be expressed as: ;in, The first one in the set of semantic principal component encoding vectors for the summary description of the data asset to be processed The semantic principal component encoding vector of the summary description of the data assets to be processed, The first one in the set of principal component encoding vectors for the classification standard semantic embedding The classification standard semantic embedding principal component encoding vector, is the number of vectors in the set of semantic principal component encoding vectors of the summary description of the data asset to be processed and the set of semantic embedding principal component encoding vectors of the classification standard, is the semantic principal component representation encoding vector of the summary description of the data asset to be processed, is the classification standard semantic embedding principal component representation coding vector; based on the data asset-classification standard semantic difference embedding compensation coding weight vector, the processing data asset summary description semantic principal component representation coding vector and the classification standard semantic embedding principal component representation coding vector are aggregated and interacted to obtain the data asset-classification standard semantic matching coding vector. The formula of this process can be expressed as: ;in, is the data asset-classification standard semantic difference embedding compensation encoding weight vector, is the semantic principal component representation encoding vector of the summary description of the data asset to be processed, is the classification standard semantic embedding principal component representation encoding vector, It is the point multiplication by position. is the point convolutional coding, is the data asset-classification standard semantic matching encoding vector.

[0056] It should be understandable that in order to more concisely and effectively reflect the overall semantic features in subsequent aggregate interactions, the positional mean vectors of the set of semantic principal component coding vectors of the summary description of the data assets to be processed and the set of classification standard semantic embedding principal component coding vectors are calculated, so as to extract common semantic features that are more representative and can better reflect the overall semantic situation, and obtain the semantic principal component representation coding vectors of the summary description of the data assets to be processed and the classification standard semantic embedding principal component representation coding vectors.

[0057] Finally, based on the data asset-classification standard semantic difference embedding compensation coding weight vector, the semantic principal component representation coding vector of the summary description of the data asset to be processed and the classification standard semantic embedding principal component representation coding vector are aggregated and interacted to utilize the difference compensation factor to dynamically adjust the fusion mode between the two, so that the final semantic matching coding vector can be adaptively adjusted according to the actual semantic difference situation, and the data asset-classification standard semantic matching coding vector is obtained.

[0058] In step S170, a set of matching probabilities is obtained based on the set of data asset-classification standard semantic matching coding vectors. Specifically, in an embodiment of the present application, a set of matching probabilities is obtained based on the set of data asset-classification standard semantic matching coding vectors, including: inputting each data asset-classification standard semantic matching coding vector in the set of data asset-classification standard semantic matching coding vectors into a matching degree explicit modeling module based on a classifier to obtain the set of matching probabilities. Correspondingly, each data asset-classification standard semantic matching coding vector in the set of data asset-classification standard semantic matching coding vectors is input into a matching degree explicit modeling module based on a classifier to obtain a set of matching probabilities. That is, the set of data asset-classification standard semantic matching coding vectors obtained by matching and compensating the semantic coding vectors of the summary description of the data asset to be processed and each classification standard semantic embedded coding vector in the set of classification standard semantic embedded coding vectors is classified, so as to further convert this semantic matching relationship into a specific matching probability by using a matching degree explicit modeling module based on a classifier, and clearly quantify this matching degree in the form of numerical values ​​to intelligently generate each matching probability.

[0059] Furthermore, the present application takes into account that the semantic coding vector of the summary description of the data asset to be processed and the semantic embedding coding vector of the classification standard respectively represent the textual semantic features of the summary description of the data asset to be processed and the classification standard in the classification standard library. When performing feature principal component compensation interaction on them, the differences in the population attributes of the text semantic features inherent in the generated data and the original data will cause fairness differences in the level of feature principal component compensation, thereby affecting the feature distribution interaction inclusiveness of each data asset-classification standard semantic matching coding vector in the set of data asset-classification standard semantic matching coding vectors, and reducing the accuracy of the matching probability obtained by inputting the classifier-based matching degree explicit modeling module.

[0060] Therefore, preferably, in an embodiment of the present application, when each data asset-classification standard semantic matching encoding vector in the set of data asset-classification standard semantic matching encoding vectors is respectively input into the classifier-based matching degree explicit modeling module to obtain a set of matching probabilities, each data asset-classification standard semantic matching encoding vector is optimized, wherein the optimization process of each data asset-classification standard semantic matching encoding vector is: determining the eigenvalue mean and eigenvalue standard deviation corresponding to the data asset-classification standard semantic matching encoding vector; multiplying the dot-wise subtraction vector of the data asset-classification standard semantic matching encoding vector and the eigenvalue mean by the eigenvalue standard deviation to obtain a first data asset-classification standard semantic matching encoding fairness target vector, And multiply the point-by-point subtraction vector of the data asset-classification standard semantic matching coding vector and the eigenvalue standard deviation by the eigenvalue mean to obtain a second data asset-classification standard semantic matching coding fair target vector; after multiplying the bit-by-bit reciprocal of the second data asset-classification standard semantic matching coding fair target vector by the first data asset-classification standard semantic matching coding fair target vector, calculate the bit-by-bit logarithm with base 2 to obtain the data asset-classification standard semantic matching coding information correction vector; take the square root of the quotient of the eigenvalue mean divided by the eigenvalue standard deviation and multiply it by the weight hyperparameter, and then multiply it by the data asset-classification standard semantic matching coding information correction vector to obtain the optimized data asset-classification standard semantic matching coding vector.

[0061] Therefore, taking into account the attribute level fairness differences of the data population corresponding to the fusion features of the data asset-classification standard semantic matching coding vector, in order to improve the interactive inclusiveness under the feature distribution diversity of the data asset-classification standard semantic matching coding vector, the crossover probability value constraint based on the data asset-classification standard semantic matching coding vector is used as the interactive fairness target representation to correct the group feature information interactive propagation of the data asset-classification standard semantic matching coding vector, and the unified statistical feature response interaction based on the data asset-classification standard semantic matching coding vector is used as the feature distribution multi-level fairness target bias to achieve a robust distribution fairness unified representation of the data asset-classification standard semantic matching coding vector, forming a fair collaboration paradigm under the feature distribution framework of the data asset-classification standard semantic matching coding vector, and improving the accuracy of the matching probability obtained by the explicit modeling module of the matching degree based on the classifier.

[0062] In step S180, based on the classification standard corresponding to the largest one in the set of matching probabilities, the category label of the data asset to be processed is determined. That is, the size of the matching probability intuitively reflects the degree of semantic fit between the data asset and each classification standard. The higher the probability, the more consistent the data asset is with the corresponding classification standard from the perspective of overall semantic matching, feature association and other comprehensive considerations. Therefore, the classification standard corresponding to the largest one in the set of matching probabilities is selected to determine the category label of the data asset, and the classification decision is made based on the objective difference in the degree of matching between the data asset and the classification standard, thereby achieving accurate and reasonable classification management of the data asset. For example, assuming that there are several classification standards of "customer information", "financial data" and "business process data", for a certain data asset to be processed, the matching probabilities with them are calculated to be 0.6, 0.3 and 0.2 respectively. Among them, the "customer information" classification standard corresponding to 0.6 is more consistent with the data asset in terms of semantics and various features, so choosing it as the basis for determining the category label can more accurately reflect the actual ownership of the data asset.

[0063] In summary, the method for automatic classification and grading of data based on the embodiment of the present application is explained, which obtains the data assets to be processed, and extracts a set of classification standards from the classification standard library, and uses data analysis and coding methods based on large models and artificial intelligence to summarize the data assets to be processed, and then semantically encodes the summarized summary description of the data assets to be processed, and at the same time, semantically embeds each classification standard, so as to intelligently generate each matching probability according to the principal component semantic matching representation between the semantic features of the summary description of the processed data assets and the semantic embedding features of each classification standard, and use the classification standard with the largest corresponding one as the category label of the data asset to be processed. The present application can summarize and semantically process data assets, extract their inherent deep meaning to intelligently create classification labels, thereby greatly reducing manual intervention, significantly improving the classification speed, and ensuring that data can respond to changes in business needs in a timely manner.

[0064] Figure 4 FIG. 1 is a block diagram of a data automatic classification and grading system according to an embodiment of the present application. Figure 4As shown, according to the data automatic classification and grading system 100 of the embodiment of the present application, it includes: a data asset acquisition module 110 to be processed, which is used to acquire the data assets to be processed; a classification standard extraction module 120, which is used to extract a set of classification standards from a classification standard library; a data asset summary description module 130, which is used to perform a data asset summary description on the data assets to be processed to obtain a summary description of the data assets to be processed; a summary description semantic encoding module 140, which is used to perform semantic encoding on the summary description of the data assets to be processed to obtain a semantic encoding vector of the summary description of the data assets to be processed; a classification standard semantic embedding module 150, which is used to perform semantic embedding on each classification standard in the set of classification standards. The module 160 is used to perform data asset semantic principal component matching compensation on the semantic encoding vector of the summary description of the data asset to be processed and each classification standard semantic embedding encoding vector in the set of the classification standard semantic embedding encoding vector to obtain a set of data asset-classification standard semantic matching encoding vectors; the module 170 is used to obtain a set of matching probabilities based on the set of data asset-classification standard semantic matching encoding vectors; the module 180 is used to determine the category label of the data asset to be processed based on the classification standard corresponding to the largest one in the set of matching probabilities.

[0065] Here, those skilled in the art will appreciate that the specific operations of each step in the above-mentioned data automatic classification and grading system have been described in detail above. Figures 1 to 3 The data automatic classification and grading method has been introduced in detail in the description of the data automatic classification and grading method, and therefore, its repeated description will be omitted.

[0066] As described above, the data automatic classification and grading system 100 according to the embodiment of the present disclosure can be implemented in various wireless terminals, such as a server with a data automatic classification and grading algorithm. In a possible implementation, the data automatic classification and grading system 100 according to the embodiment of the present disclosure can be integrated into the wireless terminal as a software module and / or a hardware module. For example, the data automatic classification and grading system 100 can be a software module in the operating system of the wireless terminal, or can be an application developed for the wireless terminal; of course, the data automatic classification and grading system 100 can also be one of the many hardware modules of the wireless terminal.

[0067] Alternatively, in another example, the data automatic classification and grading system 100 and the wireless terminal may also be separate devices, and the data automatic classification and grading system 100 may be connected to the wireless terminal via a wired and / or wireless network and transmit interactive information in accordance with an agreed data format.

[0068] The above descriptions of various implementations of the present disclosure are exemplary and not exhaustive. The disclosure is not limited to the disclosed implementations, and many modifications and variations are obvious to those of ordinary skill in the art without departing from the scope and spirit of the described implementations. The terms used herein are selected to best explain the principles of the implementations, practical applications, or improvements to the technology in the market, or to enable other persons of ordinary skill in the art to understand the various practical methods disclosed herein.

Claims

1. A method for automatic classification and grading of data, characterized in that: include: Get the data assets to be processed; extracting a set of taxonomies from a taxonomies repository; Perform a data asset summary description on the data asset to be processed to obtain a summary description of the data asset to be processed; perform semantic encoding on the summary description of the data asset to be processed to obtain a semantic encoding vector of the summary description of the data asset to be processed; perform semantic embedding encoding on each classification standard in the set of classification standards to obtain a set of classification standard semantic embedding encoding vectors; perform data asset semantic principal component matching compensation on each classification standard semantic embedding encoding vector in the set of the summary description semantic encoding vector of the data asset to be processed and the classification standard semantic embedding encoding vector to obtain a set of data asset-classification standard semantic matching encoding vectors; obtain a set of matching probabilities based on the set of data asset-classification standard semantic matching encoding vectors; determine the category label of the data asset to be processed based on the classification standard corresponding to the largest one in the set of matching probabilities.

2. The method for automatic data classification and grading according to claim 1, characterized in that: Performing a data asset summary description on the data asset to be processed to obtain a summary description of the data asset to be processed, including: inputting the data asset to be processed into a data asset summary descriptor based on a large language model to obtain the summary description of the data asset to be processed.

3. The method for automatic data classification and grading according to claim 2, characterized in that: The summary description of the data asset to be processed is semantically encoded to obtain a semantic encoding vector of the summary description of the data asset to be processed, including: segmenting the summary description of the data asset to be processed and inputting it into a semantic encoder including a word embedding layer to obtain the semantic encoding vector of the summary description of the data asset to be processed.

4. The method for automatic data classification and grading according to claim 3, characterized in that: The method comprises: performing semantic embedding coding on each classification standard in the set of classification standards to obtain a set of classification standard semantic embedding coding vectors, including: using a semantic encoder based on a Bert model to perform semantic embedding coding on each classification standard in the set of classification standards to obtain a set of classification standard semantic embedding coding vectors.

5. The method for automatic data classification and grading according to claim 4, characterized in that: Performing data asset semantic principal component matching compensation on each classification standard semantic embedded coding vector in the set of the semantic coding vector of the summary description of the data asset to be processed and the semantic embedded coding vector of the classification standard to obtain a set of data asset-classification standard semantic matching coding vectors, including: performing feature principal component analysis on the semantic coding vector of the summary description of the data asset to be processed and the semantic embedded coding vector of the classification standard to obtain a set of semantic principal component coding vectors of the summary description of the data asset to be processed and a set of semantic embedded principal component coding vectors of the classification standard; performing feature principal component analysis on the semantic coding vector of the summary description of the data asset to be processed and the semantic embedded principal component coding vector of the classification standard to obtain a set of semantic principal component coding vectors of the summary description of the data asset to be processed and a set of semantic embedded principal component coding vectors of the classification standard Reshape is performed to obtain a semantic principal component aggregation coding feature map of the summary description of the data asset to be processed and a classification standard semantic embedding principal component aggregation coding feature map; a data asset-classification standard semantic difference embedding compensation coding weight vector is calculated between the semantic principal component aggregation coding feature map of the summary description of the data asset to be processed and the classification standard semantic embedding principal component aggregation coding feature map; based on the data asset-classification standard semantic difference embedding compensation coding weight vector, a set of semantic principal component coding vectors of the summary description of the data asset to be processed and a set of semantic embedding principal component coding vectors of the classification standard are globally averaged and aggregated to obtain the data asset-classification standard semantic matching coding vector.

6. The method for automatic data classification and grading according to claim 5, characterized in that: Calculating the data asset-classification standard semantic difference embedding compensation coding weight vector between the semantic principal component aggregation coding feature map of the summary description of the data asset to be processed and the classification standard semantic embedding principal component aggregation coding feature map, including: performing semantic difference embedding on the semantic principal component aggregation coding feature map of the summary description of the data asset to be processed and the classification standard semantic embedding principal component aggregation coding feature map to obtain the semantic branch weight vector of the summary description of the data asset to be processed and the classification standard semantic embedding branch weight vector; calculating the data asset-classification standard semantic difference embedding compensation coding weight vector between the semantic branch weight vector of the summary description of the data asset to be processed and the classification standard semantic embedding branch weight vector.

7. The method for automatic data classification and grading according to claim 6, characterized in that: Based on the data asset-classification standard semantic difference embedding compensation coding weight vector, the set of semantic principal component coding vectors of the summary description of the data asset to be processed and the set of semantic embedding principal component coding vectors of the classification standard are globally averaged and aggregated to obtain the data asset-classification standard semantic matching coding vector, including: calculating the positional mean vector of the set of semantic principal component coding vectors of the summary description of the data asset to be processed and the set of semantic embedding principal component coding vectors of the classification standard to obtain the semantic principal component representation coding vector of the summary description of the data asset to be processed and the classification standard semantic embedding principal component representation coding vector; based on the data asset-classification standard semantic difference embedding compensation coding weight vector, the semantic principal component representation coding vector of the summary description of the data asset to be processed and the classification standard semantic embedding principal component representation coding vector are aggregated and interacted to obtain the data asset-classification standard semantic matching coding vector.

8. The method for automatic data classification and grading according to claim 7, characterized in that: Based on the set of data asset-classification standard semantic matching coding vectors, a set of matching probabilities is obtained, including: inputting each data asset-classification standard semantic matching coding vector in the set of data asset-classification standard semantic matching coding vectors into a classifier-based matching degree explicit modeling module to obtain the set of matching probabilities.

9. A data automatic classification and grading system, characterized in that: include: A module for acquiring data assets to be processed, used to acquire data assets to be processed; A classification standard extraction module, used for extracting a set of classification standards from a classification standard library; A data asset summary description module, used for performing a data asset summary description on the data asset to be processed to obtain a summary description of the data asset to be processed; a summary description semantic coding module, used for performing semantic coding on the summary description of the data asset to be processed to obtain a semantic coding vector of the summary description of the data asset to be processed; a classification standard semantic embedding module, used for performing semantic embedding coding on each classification standard in the set of classification standards to obtain a set of classification standard semantic embedding coding vectors; a semantic principal component matching module, used for performing data asset semantic principal component matching compensation on each classification standard semantic embedding coding vector in the set of the data asset summary description semantic coding vector and the classification standard semantic embedding coding vector to obtain a set of data asset-classification standard semantic matching coding vectors; A matching probability generation module is used to obtain a set of matching probabilities based on a set of semantic matching encoding vectors of the data asset and classification standard; a category label determination module is used to determine the category label of the data asset to be processed based on the classification standard corresponding to the largest one in the set of matching probabilities.

Images