Method, device and medium for subject classification of data assets
By combining a dual classification model with deep learning technology, the problems of low efficiency and high cost in data asset subject classification are solved, and fast and accurate automated intelligent classification is achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- AGRICULTURAL BANK OF CHINA
- Filing Date
- 2022-10-28
- Publication Date
- 2026-07-10
AI Technical Summary
Existing methods for classifying data assets by topic rely on manual operation, which is inefficient and costly. Intelligent classification systems are complex and difficult to maintain.
A dual classification model is adopted, which is trained based on the metadata of the data asset and the relationship between the parent data asset and the topic category. The final category is determined by combining the results of the first and second topic classifications. Deep learning models such as FastText and BERT are used for training.
It enables rapid, accurate, automated, and intelligent classification of data assets, reducing system complexity and maintenance costs while improving classification accuracy and efficiency.
Smart Images
Figure CN115618264B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of data processing technology, and in particular to a method, apparatus, device, and medium for subject classification of data assets. Background Technology
[0002] In the era of big data, as enterprises accumulate data, data is playing an increasingly important role and has become a crucial asset for businesses. The financial sector possesses substantial data assets, and enterprises' data assets are growing daily. Effectively organizing data according to certain standards to form a systematic data service catalog will facilitate data asset management, build a bridge between technology and business, and provide convenience for business personnel to find valuable data.
[0003] In existing technologies, most data asset subject categories are manually defined. However, manual subject classification is labor-intensive, inefficient, and requires experienced professionals. Furthermore, existing intelligent subject classification methods train models separately at each parent node of the asset subject classification tree. Each model contains primary and secondary classification models, making the overall asset subject classification system complex, costly to maintain, and involving a large number of model parameters that need to be adjusted.
[0004] Therefore, it is very important to be able to quickly and accurately classify topics. Summary of the Invention
[0005] This invention provides a method, apparatus, device, and medium for the subject classification of data assets, so as to achieve rapid and accurate automated intelligent classification of the subjects of data assets.
[0006] According to one aspect of the present invention, a method for subject classification of data assets is provided, the method comprising:
[0007] Obtain reference data for the data assets to be classified;
[0008] Based on the first classification model, a first topic classification result of the data asset to be classified is obtained according to the reference data; wherein, the first classification model is pre-trained based on the relationship between the metadata of the data asset and the topic category;
[0009] Based on the second classification model, a second topic classification result of the data asset to be classified is obtained according to the reference data; wherein, the second classification model is pre-trained based on the relationship between the data asset's parent data asset and the topic category;
[0010] The final subject category of the data asset to be classified is determined based on the first subject classification result and the second subject classification result.
[0011] According to another aspect of the present invention, a data asset subject classification device is provided, the device comprising:
[0012] The data acquisition module is used to acquire reference data for the data assets to be classified.
[0013] The first classification result determination module is used to obtain the first topic classification result of the data asset to be classified based on the reference data and the first classification model; wherein, the first classification model is pre-trained based on the relationship between the metadata of the data asset and the topic categories;
[0014] The second classification result determination module is used to obtain the second topic classification result of the data asset to be classified based on the reference data and the second classification model; wherein, the second classification model is pre-trained based on the relationship between the data asset's parent data asset and the topic category;
[0015] The topic category determination module is used to determine the final topic category of the data asset to be classified based on the first topic classification result and the second topic classification result.
[0016] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:
[0017] At least one processor; and
[0018] A memory communicatively connected to the at least one processor; wherein,
[0019] The memory stores a computer program that can be executed by the at least one processor, which enables the at least one processor to perform the data asset subject classification method according to any embodiment of the present invention.
[0020] According to another aspect of the present invention, a computer-readable storage medium is provided, the computer-readable storage medium storing computer instructions for causing a processor to execute and implement the data asset subject classification method according to any embodiment of the present invention.
[0021] The technical solution of this invention includes: acquiring reference data of the data asset to be classified; obtaining a first topic classification result of the data asset to be classified based on the reference data using a first classification model; obtaining a second topic classification result of the data asset to be classified based on the reference data using a second classification model; and finally determining the final topic category of the data asset to be classified based on the first and second topic classification results. This technical solution obtains accurate topic classification results by inputting the reference data of the data asset to be classified into a first classification model trained based on the relationship between the metadata of the data asset and topic categories, and a second classification model trained based on the relationship between the parent data asset of the data asset and topic categories. Then, by analyzing the topic classification results, an accurate final topic category is obtained. This avoids inaccurate determination of the final topic category due to the inaccuracy of a single model, and achieves rapid and accurate automated intelligent classification of the topics of data assets.
[0022] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description
[0023] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0024] Figure 1 This is a flowchart of a data asset subject classification method provided in Embodiment 1 of the present invention;
[0025] Figure 2 This is a structural diagram of the target classification data assets applicable to the embodiments of the present invention;
[0026] Figure 3 This is a schematic diagram of the structure of a data asset subject classification device provided in Embodiment 3 of the present invention;
[0027] Figure 4 This is a schematic diagram of the structure of an electronic device that implements the data asset subject classification method of this invention. Detailed Implementation
[0028] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.
[0029] It should be noted that the terms "first," "second," and "target," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.
[0030] Example 1
[0031] Figure 1 This is a flowchart illustrating a data asset subject classification method according to Embodiment 1 of the present invention. This embodiment is applicable to the classification of data asset subjects. The method can be executed by a data asset subject classification device, which can be implemented in hardware and / or software. This data asset subject classification device can be configured in an electronic device that has a data asset subject classification method. Figure 1 As shown, the method includes:
[0032] S110. Obtain reference data for the data assets to be classified.
[0033] The data assets to be classified can be asset data that has not been subject to a specific category. Reference data can be information from the data assets to be classified that can represent the characteristics of the data assets.
[0034] Specifically, as enterprises continue to develop, their data also accumulates. This data is an important asset for enterprises. In order to facilitate the management of this data asset, it is necessary to classify it so that enterprise personnel can find it. Before classifying the data asset, it is necessary to acquire the unclassified data asset, that is, the data asset to be classified, and determine the characteristic information of the data asset to be classified, that is, the reference data, so that the subject category of the data asset to be classified can be determined through subsequent analysis of the reference data.
[0035] S120. Based on the first classification model, obtain the first topic classification result of the data asset to be classified according to the reference data; wherein, the first classification model is pre-trained based on the relationship between the metadata of the data asset and the topic category.
[0036] The reference data can be feature data adapted to the first classification model. Metadata includes, but is not limited to, table names, all field names, table comments, field comments, etc.; metadata information can be obtained from data production environments or metadata management systems, etc., where metadata can be acquired. The topic categories are accurate labels obtained through manual classification and verification. The first topic classification result includes the first topic category and the first confidence score. The first confidence score describes the strength of the association between the first topic category and the data asset to be classified, that is, the credibility of the first topic category as the topic category of the data asset to be classified. The first classification model is a deeply trained model.
[0037] Specifically, feature data adapted to the first classification model from the data assets to be classified is obtained, i.e., reference data, so that the first classification model can analyze and predict the reference data to determine the accurate classification result of the data assets to be classified, and use it as the first subject classification result.
[0038] In one feasible embodiment, optionally, the training process of the first classification model is as follows: steps A1-A3:
[0039] Step A1: Obtain the metadata of the target data asset.
[0040] Step A2: Use the metadata of the target data asset as features and the target topic category of the target data asset as labels to obtain the first training data.
[0041] Step A3: Input the first training data into the pre-built first classification model for training until the model converges, and obtain the first classification model based on metadata.
[0042] The target data asset can be a data asset with a known topic type, which can be determined through historical classification or by human classification. The first training data includes the metadata of the target data asset and the target topic category of the target data asset, which can reflect the mapping relationship between the metadata of the target data asset and the target topic category of the target data asset.
[0043] Specifically, the process involves acquiring the target data assets. Since the English table and field names of these assets are typically abbreviations of English letters, which lack expressive power, the Chinese table names, field names, table comments, and field comments are used first. The target data assets are then analyzed and preprocessed to accurately obtain their metadata in a text format, organized according to a specific structure. For example, metadata text separated by commas, spaces, or other delimiters is obtained. Preprocessing includes the standard preprocessing workflow for Chinese text classification: word segmentation and stop word removal. Chinese word segmentation refers to breaking down a sequence of Chinese characters into individual words. Stop words refer to non-semantic words or other words without practical function in natural language, such as modifiers and auxiliary words. Stop word removal can accelerate model convergence and improve the model's ability to distinguish keywords. Commonly used tools in the industry can be used for Chinese word segmentation, such as the jieba package in Python. Since there is no unified standard for stop words, a stop word removal table specific to this solution can be set up, and an automated script can be used to remove stop words from the training data one by one.
[0044] After obtaining the metadata of the target data asset, the target topic category of the target data asset is determined. Then, the metadata of the target data asset is used as a feature, and the target topic category of the target data asset is used as a label to accurately determine the mapping relationship between the metadata and the target category. This allows the metadata and the target topic category to be used as the first training data and input into the pre-built first classification model, ultimately obtaining an accurate first classification model that can determine the topic category of the data asset to be classified.
[0045] This technical solution analyzes and processes the target data assets to accurately determine their metadata. Then, it uses the metadata as features and the target topic category as a label to obtain accurate first training data. This allows the first training data to be trained using a pre-built first classification model, thereby obtaining a representative first classification model based on metadata. This achieves accurate determination of the first classification model and ensures the reliability of the model.
[0046] In one feasible embodiment, optionally, based on the first classification model, a first topic classification result of the data asset to be classified is obtained according to the reference data, including:
[0047] The metadata of the data asset to be classified is input into the first classification model based on the metadata to obtain the first topic classification result of the data asset to be classified.
[0048] The reference data includes metadata about the data assets to be classified. The first topic classification result includes the first topic category and the first confidence level.
[0049] Specifically, the metadata of the data asset to be classified is obtained and input into the first classification model to accurately obtain the first topic classification result, which includes the first topic category and the first confidence level. This enables the accurate determination of the first topic category of the data asset to be classified based on the first classification model, as well as the reliability of the first topic category determined by the first classification model. Subsequently, the topic classification of the data asset to be classified can be accurately determined by analyzing the first topic classification result.
[0050] This technical solution achieves accurate determination of the first subject classification result of the data assets to be classified by accurately analyzing the metadata of the data assets to be classified through the first classification model.
[0051] S130. Based on the second classification model, obtain the second topic classification result of the data asset to be classified according to the reference data; wherein, the second classification model is pre-trained based on the relationship between the data asset's parent data asset and the topic category.
[0052] The reference data can be feature data adapted to the second classification model. The second topic classification result includes a second topic category and a second confidence score; the second confidence score describes the strength of the association between the second topic category and the data asset to be classified, that is, the credibility of the second topic category as the topic category of the data asset to be classified. The second classification model is a deeply trained model.
[0053] Specifically, feature data adapted to the second classification model from the data assets to be classified is obtained, i.e., reference data, so that the second classification model can analyze and predict the reference data to determine the accurate classification result of the data assets to be classified, and use it as the second subject classification result.
[0054] In one feasible embodiment, optionally, the training process of the second classification model is as follows: steps B1-B3:
[0055] Step B1: Obtain the dependent metadata of the target data asset; wherein the dependent metadata is determined based on the parent data asset of the target data asset.
[0056] Step B2: Use the dependent metadata and the parent topic category of the parent data asset as features, and the target topic category of the target data asset as a label to obtain the second training data.
[0057] Step B3: Input the second training data into the pre-built second classification model to obtain a second classification model based on the upper-level data assets.
[0058] The target data asset can be obtained by selecting, inserting, or associating certain fields of the parent data asset, see [link to relevant documentation]. Figure 2 The parent data asset can exist before the target data asset is established. The relationship between dependency metadata and parent topic categories is already determined, and the parent topic categories can be obtained through manual annotation or model inference. The second training data includes dependency metadata, the parent topic categories of the parent data asset, and the target topic categories of the target data asset, used to describe the mapping relationship between dependency metadata and the parent topic categories of the parent data asset and the target topic categories of the target data asset. Dependency metadata is metadata information such as table names, field names, table comments, and field comments of the parent data asset.
[0059] Specifically, the target data asset is acquired and analyzed to determine the dependent metadata and the parent topic category of the parent data asset. The dependent metadata and the parent topic category of the parent data asset are used as features, and the target topic category of the target data asset is used as a label. This accurately determines the mapping relationship between the dependent metadata and the parent topic category of the parent data asset and the target topic category of the target data asset. This allows the dependent metadata, the parent topic category of the parent data asset, and the target topic category of the target data asset to be used as second training data and input into a pre-built second classification model. As a result, an accurate second classification model can be obtained to determine the topic category of the data asset to be classified.
[0060] This technical solution analyzes and processes the target data asset and its parent data asset to accurately determine the dependent metadata in the target data asset and the parent topic category of the parent data asset. Then, it uses the dependent metadata of the target data asset as features and the target topic category of the target data asset as labels to obtain accurate second training data. This allows the second training data to be trained using a pre-built second classification model, thereby obtaining a representative second classification model based on the parent data asset. This achieves accurate determination of the second classification model and ensures the reliability of the model.
[0061] In one feasible embodiment, optionally, based on the second classification model, a second topic classification result for the data asset to be classified is obtained according to the reference data, including:
[0062] The dependent metadata of the data asset to be classified and the topic category of the parent data asset of the data asset to be classified are input into the second classification model based on the parent data asset to obtain the second topic classification result of the data asset to be classified.
[0063] The reference data includes the dependent metadata of the data asset to be classified and the topic categories of the parent data asset of the data asset to be classified. The second topic classification result includes the second topic category and the second confidence level.
[0064] Specifically, the dependent metadata of the data asset to be classified and the topic categories of its parent data asset are obtained. These are then input into the second classification model to accurately obtain a second topic classification result that includes the second topic category and the second confidence level. This allows for the accurate determination of the second topic category of the data asset to be classified based on the second classification model, as well as the reliability of the second topic category determined by the second classification model. Subsequently, by analyzing the second topic classification result, the topic classification of the data asset to be classified can be accurately determined.
[0065] This technical solution uses a second classification model to accurately analyze the dependent metadata of the data asset to be classified and the subject categories of the parent data asset of the data asset to be classified, thereby achieving accurate determination of the second subject classification result of the data asset to be classified.
[0066] S140. Determine the final theme category of the data asset to be classified based on the first theme classification result and the second theme classification result.
[0067] Specifically, the first and second topic classification results are obtained, and the first and second topic classification results are analyzed and processed. The final topic category of the data asset to be classified is determined by the results of the analysis and processing, thus avoiding inaccurate topic category determination due to the inaccuracy of a single result.
[0068] In a feasible embodiment, optionally, the first topic classification result and the second topic classification result can be analyzed as follows to obtain the accurate final topic category of the data asset to be classified: if the first topic category in the first topic classification result and the second topic category in the second topic classification result are the same, then the first topic category is determined as the final topic category of the data asset to be classified; if the first topic category in the first topic classification result and the second topic category in the second topic classification result are different, then the final topic category of the data asset to be classified is determined according to the first confidence level in the first topic classification result and the second confidence level in the second topic classification result.
[0069] This technical solution combines the first topic classification result obtained based on the first classification model and the second topic classification result obtained based on the second classification model for analysis. This avoids errors in topic classification results caused by the inaccuracy of a single model, which would affect the determination of the final topic category of the data asset to be classified, and achieves accurate determination of the topic category of the data asset to be classified.
[0070] This application's technical solution obtains accurate topic classification results by inputting reference data of the data assets to be classified into a first classification model trained based on the relationship between the data asset's metadata and topic categories, and a second classification model trained based on the relationship between the data asset's parent data assets and topic categories. Then, by analyzing the topic classification results, an accurate final topic category is obtained. This avoids inaccuracies in the final topic category determination due to the inaccuracy of a single model, achieving rapid and accurate automated intelligent classification of data asset topics. Furthermore, the deep learning-based model eliminates tedious feature engineering, resulting in low system complexity and good scalability.
[0071] Example 2
[0072] This embodiment provides a detailed description of the pre-trained models for the first and second classification models described in the above embodiments.
[0073] Optionally, the first and second classification models are trained based on the FastText model or the BERT pre-trained model.
[0074] Optionally, for the FastText model: FastText is a text classification tool open-sourced by Facebook in 2016. It has a fast training speed and word vector training capabilities. Therefore, using the FastText model can eliminate the need to separately obtain word vectors. The input to the FastText model is a text file, where each line of the file is a training data point. Each training data point is a label for a data asset and the metadata text of the data asset after word segmentation and stop word removal preprocessing. For example, __label__ is a prefix for the label, such as: "__label__ Customer Legal Person Sued Historical Financial Entity Number Customer Number Record Number Plaintiff Name Reason for Sue Currency Code Amount Sued Date Judgment Identifier Judgment Execution Currency Code Judgment Execution Amount Judgment Execution Date Judgment Execution Result Deletion Flag Institution Code Operator Code Operation Date System Source Code Last Update Date Start Date End Date Valid Date Timestamp".
[0075] Among them, "customer" is the label of this training data, and the remaining part "legal person being sued..." is the feature of the training data.
[0076] Training is done using the `train_supervised` method of FastText. Assuming the model name is defined as `model`, calling the `model.test` method to test the validation set will yield the model's precision and recall on the validation set. By appropriately adjusting hyperparameters such as the number of training epochs and the learning rate, the model can achieve the highest precision and recall on the validation set; this model is then considered well-trained.
[0077] For example, to implement a first-class classification model using the FastText model, the trained model can be saved using the `model.save` method, and the saved model can be loaded using the `load_model` method. The model prediction method is `model.predict`, which takes as input unlabeled metadata of the data asset after word segmentation and stop word removal preprocessing, and outputs the topic category of the data asset and its confidence score.
[0078] Optionally, for the BERT pre-trained model: BERT (Bidirectional Encoder Representation from Transformers) is a pre-trained language representation model derived from Transformer. The model structure of BERT is the encoder part of Transformer. It can be understood that when given a text, the output of BERT is the word vector (or word embedding) of that text.
[0079] The purpose of BERT training is to learn the word embeddings of text, and it trains the model based on two tasks.
[0080] Task 1's concept is a "fill-in-the-blank" problem. The input is a sentence where 15% of the words at random positions are replaced by a fixed special symbol [MASK] or another random word. The model's output is a sequence of the same length as the input, where each element is a vector, corresponding one-to-one with each word in the input. The output vectors corresponding to the replaced unknown words are the word vectors predicted by the model for those unknown words. These word vectors are then subjected to a linear transformation and a softmax function to obtain another vector, the softmax output. The value at each position in this vector represents the probability distribution of the unknown word being a given character. The BERT model's loss function is the cross-entropy between the softmax probability of the unknown word and the vector corresponding to the true word. Minimizing this loss function yields the word embedding vector for the unknown word.
[0081] Task 2 is next sentence prediction. The input consists of two sentences. The first sentence begins with a special category label [CLS], and the two sentences are separated by a special label [SEP]. The model's output is the BERT output corresponding to the [CLS] label position. This output is processed by a linear model to determine whether the second sentence is the next sentence after the first. The model's training process involves understanding the relationship between the two sentences.
[0082] When using the BERT pre-trained model in practice, only the BERT module is retained, and the linear transformation and softmax parts used in training are discarded. That is, the output of BERT is the word embedding vector of the input sequence.
[0083] Furthermore, BERT-based pre-trained models can be spliced together with linear models, RNN models such as LSTM and GRU, CNN models, attention-based models, and any other models that can be used for data asset classification.
[0084] For example, the first classification model uses BERT's output on the first special label [CLS]. A pre-trained BERT model is used, and fine-tuning is performed on top of this. The model's input consists of the special label [CLS] and metadata of the data asset (the input is tokenized, and the tokens are converted into indices before being fed into the model). The output is the topic category of the data asset. Unlike the pre-training stage, the fine-tuning process is a supervised learning process. The output corresponding to [CLS] passes through a linear model and a softmax layer to obtain the probability distribution of the data asset's topic category, i.e., the confidence level. The cross-entropy between this probability distribution and the topic category of the data asset after one-hot encoding is used as the loss function. Minimizing this loss function yields the prediction result for the data asset's classification.
[0085] Optionally, deep learning-based text classification methods can implement both primary and secondary classification models, including but not limited to TextCNN, TextRNN, attention-based methods, and GPT-based pre-trained models.
[0086] This technical solution describes in detail how to train the target classification data asset using a FastText model or a BERT pre-trained model to obtain an accurate first classification model and a second classification model. This achieves accurate model determination and facilitates the subsequent input of reference data of the data asset to be classified into the model to accurately classify the data asset.
[0087] Example 3
[0088] Figure 3This is a schematic diagram of a data asset subject classification device according to Embodiment 3 of the present invention. Figure 3 As shown, the device includes:
[0089] The data acquisition module is used to acquire reference data for the data assets to be classified.
[0090] The first classification result determination module is used to obtain the first topic classification result of the data asset to be classified based on the reference data and the first classification model; wherein, the first classification model is pre-trained based on the relationship between the metadata of the data asset and the topic categories;
[0091] The second classification result determination module is used to obtain the second topic classification result of the data asset to be classified based on the reference data and the second classification model; wherein, the second classification model is pre-trained based on the relationship between the data asset's parent data asset and the topic category;
[0092] The topic category determination module is used to determine the final topic category of the data asset to be classified based on the first topic classification result and the second topic classification result.
[0093] Optionally, the second classification result determination module includes a second model training unit, specifically used for:
[0094] Obtain the dependent metadata of the target data asset; wherein the dependent metadata is determined based on the parent data asset of the target data asset;
[0095] The second training data is obtained by using the dependent metadata and the parent topic category of the parent data asset as features, and the target topic category of the target data asset as a label;
[0096] The second training data is input into a pre-built second classification model to obtain a second classification model based on the upper-level data assets.
[0097] Optional, the second classification result determination module is specifically used for:
[0098] The dependent metadata of the data asset to be classified and the topic category of the parent data asset of the data asset to be classified are input into the second classification model based on the parent data asset to obtain the second topic classification result of the data asset to be classified.
[0099] The reference data includes the dependent metadata of the data asset to be classified and the subject category of the parent data asset of the data asset to be classified.
[0100] Optionally, the first classification result determination module includes a first model training unit, specifically used for:
[0101] Obtain the metadata of the target data asset;
[0102] The first training data is obtained by using the metadata of the target data asset as features and the target topic category of the target data asset as labels;
[0103] The first training data is input into a pre-built first classification model to obtain a first classification model based on metadata.
[0104] Optional, the first classification result determination module is specifically used for:
[0105] The metadata of the data asset to be classified is input into the first classification model based on the metadata to obtain the first topic classification result of the data asset to be classified.
[0106] The reference data includes the metadata of the data asset to be classified.
[0107] Optional, a topic category determination module, specifically used for:
[0108] If the first topic category in the first topic classification result is the same as the second topic category in the second topic classification result, then the first topic category is determined as the final topic category of the data asset to be classified.
[0109] If the first topic category in the first topic classification result is different from the second topic category in the second topic classification result, then the final topic category of the data asset to be classified is determined based on the first confidence level in the first topic classification result and the second confidence level in the second topic classification result.
[0110] The topic classification results include topic category and confidence level.
[0111] Optionally, the first classification model and the second classification model are trained based on the FastText model or the BERT pre-trained model.
[0112] The data asset subject classification device provided in the embodiments of the present invention can execute the data asset subject classification method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the method execution.
[0113] The acquisition, storage, use, and processing of data in this application comply with relevant national laws and regulations and do not violate public order and good morals.
[0114] Example 4
[0115] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.
[0116] Figure 4 A schematic diagram of an electronic device is shown that can be used to implement the data asset subject classification method of embodiments of the present invention. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (such as helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.
[0117] like Figure 4 As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12 or a random access memory (RAM) 13, communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer program stored in the ROM 12 or loaded from storage unit 18 into the RAM 13. The RAM 13 may also store various programs and data required for the operation of the electronic device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.
[0118] Multiple components in electronic device 10 are connected to I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of displays, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.
[0119] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as the subject classification method for data assets.
[0120] In some embodiments, the data asset subject classification method may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the data asset subject classification method described above may be performed. Alternatively, in other embodiments, processor 11 may be configured to perform the data asset subject classification method by any other suitable means (e.g., by means of firmware).
[0121] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.
[0122] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.
[0123] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.
[0124] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).
[0125] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or computing systems that include middleware components (e.g., application servers), or computing systems that include frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.
[0126] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.
[0127] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.
[0128] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.
Claims
1. A method for thematic classification of data assets, characterized in that, include: Obtain reference data for the data asset to be classified; wherein, the reference data includes the metadata of the data asset to be classified, the dependent metadata of the data asset to be classified, and the subject category of the parent data asset of the data asset to be classified; the metadata includes table name, all field names, table comments, and field comments; the dependent metadata is the metadata information of the parent data asset; Based on the first classification model, a first topic classification result of the data asset to be classified is obtained according to the reference data; wherein, the first classification model is pre-trained based on the relationship between the metadata of the data asset and the topic category; Based on the second classification model, a second topic classification result of the data asset to be classified is obtained according to the reference data; wherein, the second classification model is pre-trained based on the relationship between the data asset's parent data asset and topic categories; The final theme category of the data asset to be classified is determined based on the first theme classification result and the second theme classification result; The training process of the first classification model includes: acquiring target data assets; analyzing and preprocessing the target data assets to obtain metadata of the target data assets; using the metadata of the target data assets as features and the target topic category of the target data assets as labels to obtain first training data; and inputting the first training data into a pre-built first classification model to obtain a first classification model based on metadata. The preprocessing includes a standard preprocessing procedure for Chinese text classification, which involves word segmentation and stop word removal. The step of obtaining the first theme classification result of the data asset to be classified based on the reference data according to the first classification model includes: inputting the metadata of the data asset to be classified into the first classification model constructed based on the metadata to obtain the first theme classification result of the data asset to be classified. The training process of the second classification model includes: acquiring the dependency metadata of the target data asset; using the dependency metadata and the parent topic category of the parent data asset as features, and the target topic category of the target data asset as a label, to obtain second training data; inputting the second training data into a pre-built second classification model to obtain a second classification model based on the parent data asset; wherein the dependency metadata is determined according to the parent data asset of the target data asset. The step of obtaining the second topic classification result of the data asset to be classified based on the reference data according to the second classification model includes: inputting the dependent metadata of the data asset to be classified and the topic category of the parent data asset of the data asset to be classified into the second classification model constructed based on the parent data asset to obtain the second topic classification result of the data asset to be classified.
2. The method according to claim 1, characterized in that, The topic classification results include topic category and confidence level; Accordingly, the final topic category of the data asset to be classified is determined based on the first topic classification result and the second topic classification result, including: If the first topic category in the first topic classification result is the same as the second topic category in the second topic classification result, then the first topic category is determined as the final topic category of the data asset to be classified. If the first topic category in the first topic classification result is different from the second topic category in the second topic classification result, then the final topic category of the data asset to be classified is determined based on the first confidence level in the first topic classification result and the second confidence level in the second topic classification result.
3. The method according to claim 1, characterized in that, The first classification model and the second classification model are trained based on the fasttext model or the BERT pre-trained model.
4. A data asset subject classification device, characterized in that, include: The data acquisition module is used to acquire reference data for the data assets to be classified; wherein, the reference data includes the metadata of the data assets to be classified, the dependent metadata of the data assets to be classified, and the subject category of the parent data assets of the data assets to be classified; the metadata includes table name, all field names, table comments, and field comments; the dependent metadata is the metadata information of the parent data assets; The first classification result determination module is used to obtain the first topic classification result of the data asset to be classified based on the reference data and the first classification model; wherein, the first classification model is pre-trained based on the relationship between the metadata of the data asset and the topic categories; The second classification result determination module is used to obtain the second topic classification result of the data asset to be classified based on the reference data and the second classification model; wherein, the second classification model is pre-trained based on the relationship between the data asset's parent data asset and the topic category; The topic category determination module is used to determine the final topic category of the data asset to be classified based on the first topic classification result and the second topic classification result; The first classification result determination module includes a first model training unit; The first model training unit is configured to: acquire target data assets; analyze and preprocess the target data assets to obtain metadata of the target data assets; use the metadata of the target data assets as features and the target topic category of the target data assets as labels to obtain first training data; input the first training data into a pre-built first classification model to obtain a first classification model based on metadata; wherein, the preprocessing includes a standard preprocessing procedure for Chinese text classification; the standard preprocessing procedure is word segmentation and stop word removal; The first classification result determination module is specifically used to: input the metadata of the data asset to be classified into the first classification model constructed based on the metadata, and obtain the first topic classification result of the data asset to be classified; The second classification result determination module includes a second model training unit; The second model training unit is configured to: acquire the dependency metadata of the target data asset; use the dependency metadata and the parent topic category of the parent data asset as features, and use the target topic category of the target data asset as a label to obtain second training data; input the second training data into a pre-built second classification model to obtain a second classification model based on the parent data asset; wherein the dependency metadata is determined according to the parent data asset of the target data asset; The second classification result determination module is specifically used to: input the dependent metadata of the data asset to be classified and the topic category of the parent data asset of the data asset to be classified into the second classification model constructed based on the parent data asset, and obtain the second topic classification result of the data asset to be classified.
5. An electronic device, characterized in that, The electronic device includes: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the subject classification method for data assets according to any one of claims 1-3.
6. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that cause a processor to execute the subject classification method for data assets according to any one of claims 1-3.