Intelligent data cataloging method and system for heterogeneous data guided cross-modal fusion and continual learning

Abstract

This invention belongs to the fields of data governance and artificial intelligence technology, and discloses an intelligent data cataloging method and system for cross-modal fusion and continuous learning guided by heterogeneous data. The method encodes structured metadata, unstructured text, and unstructured images into structured pattern vectors, text content vectors, and image visual vectors, respectively; it performs cross-modal attention fusion using the structured pattern vector as the query vector and the combined vectors formed by concatenating the other two along the feature dimensions as key-value vectors; it classifies the fused semantic tensor through a classification model; when learning new categories, it constructs a parameter update offset penalty term based on the diagonal elements of the Fisher information matrix of each parameter to update the parameter offset; and it uses the classification results as anchors to determine the business subject domain, sensitivity level, and field-level data lineage to form multi-dimensional cataloging information. This invention improves cross-modal semantic alignment accuracy, maintains the performance of old categories, and reduces manual intervention in multi-dimensional cataloging.

Description

Technical Field

[0001] This invention belongs to the fields of data governance and artificial intelligence technology, and relates to an intelligent data cataloging method and system that guides cross-modal fusion and continuous learning based on heterogeneous data. Background Technology

[0002] As enterprises continue their digital transformation, the data generated by business systems is becoming increasingly multi-source, heterogeneous, and massive, encompassing various formats such as relational databases, unstructured documents, images, and message queues. Data cataloging, as a fundamental step in data governance, aims to establish a unified view of data assets, clarifying the business semantics, sources, destinations, attribute characteristics, and management standards of the data, providing support for data reuse, data security, and data analysis. Existing solutions for automated cataloging of multi-source heterogeneous data can be broadly categorized as follows, each with its own engineering shortcomings: 1. One approach is to process structured metadata and unstructured content separately: structured metadata such as field names, field comments, and table structures are categorized using rule-based or dictionary-based methods, while unstructured documents and images are processed using separate text classification or image recognition models. This approach ignores the strong semantic relationships that commonly exist between structured fields and unstructured content. For example, a structured field name "Customer Number" may belong to the same semantic category as "Customer ID" described in a business document or a customer identifier identified in a surveillance image. It is difficult to establish reliable semantic alignment between the structured and unstructured results during the cataloging stage, resulting in low cataloging accuracy.

[0003] 2. Another approach is to encode the structured metadata and unstructured content separately before performing feature concatenation or simple weighted fusion. While this approach formally merges the features from both sides into the same vector space, it lacks a semantic guidance mechanism dominated by the structured metadata. During the fusion stage, the model struggles to focus on local details in the unstructured content that are semantically related to the structured fields, and there is insufficient weight suppression of irrelevant background content, resulting in limited cross-modal semantic alignment accuracy.

[0004] 3. Existing technologies mostly adopt the method of periodically retraining static models to deal with new data categories, which is costly in terms of computing power and has a long iteration cycle; while conventional incremental learning methods are difficult to maintain stable classification accuracy in complex enterprise-level cataloging scenarios.

[0005] 4. Most existing automated cataloging solutions only complete the classification and identification of data categories. They lack the ability to automatically deduce cataloging dimensions such as business subject domain affiliation, sensitivity level marking, and field-level data lineage. They often rely on a lot of manual completion, resulting in single cataloging dimensions and low standardization, which makes it difficult to provide complete support for data security control and data asset management.

[0006] To address the aforementioned issues, there is an urgent need for an intelligent data cataloging technology that can achieve cross-modal feature depth alignment using structured metadata as a guiding signal, protect existing classification knowledge while continuously learning new data categories, and determine multi-dimensional cataloging information based on classification results. Summary of the Invention

[0007] The purpose of this invention is to overcome the problems of cross-modal semantic alignment difficulties, model solidification and difficulty in maintaining the performance of old categories under new task training, and single cataloging dimension in the prior art, and to provide an intelligent data cataloging method and system that guides cross-modal fusion and continuous learning with heterogeneous data.

[0008] To address the aforementioned issues, this invention employs structured pattern vectors as query vectors during the fusion phase and combined vectors formed by concatenating unstructured text and unstructured images along feature dimensions as key and value vectors, respectively. During the classification phase, it introduces parameters as an importance indicator of the classification performance of previous tasks to apply regularization constraints to parameter update offsets. Finally, during the cataloging output phase, it uses the classification results as anchor points to determine three cataloging dimensions: business subject domain, sensitivity level, and field-level data lineage.

[0009] To achieve the above-mentioned objectives, the technical solution provided by this invention includes: Intelligent data cataloging methods guided by heterogeneous data, involving cross-modal fusion and continuous learning, include: The structured metadata, unstructured text data, and unstructured image data in the multi-source heterogeneous data are encoded respectively to obtain structured pattern vectors, text content vectors, and image visual vectors. Using the structured pattern vector as the query vector, and the combined vector formed by concatenating the text content vector and the image visual vector along the feature dimension as the key vector and value vector respectively, cross-modal attention fusion is performed to obtain the fused semantic tensor; Using the fused semantic tensor as input, the multi-source heterogeneous data is cataloged and classified through a classification model to obtain the classification result; When the classification model learns new data categories, the diagonal elements of the Fisher information matrix of the loss function of the classification model with respect to each parameter after the old task training are used as importance indicators. The parameter update offset of each parameter is weighted according to the importance indicators to obtain the parameter update offset penalty term. The training objective of the new task is constructed as a weighted combination of the new task classification loss term and the parameter update offset penalty term. Using the classification results as anchors, the business subject domain, sensitivity level, and field-level data lineage of the multi-source heterogeneous data are determined to form multi-dimensional cataloging information.

[0010] Preferably, the encoding of the structured metadata and the encoding of the unstructured text data are achieved through a common pre-trained language model, where common origin means that they share a basic vocabulary and are pre-trained based on the same pre-trained corpus; wherein the structured metadata is encoded by concatenating field names, field annotations and the structure information of the table to which it belongs into a short text according to a preset template and then inputting it into the common pre-trained language model; the encoding of the unstructured image data is achieved through a convolutional neural network.

[0011] Preferably, the step of weighting the parameter update offset of each parameter according to the importance index to obtain the parameter update offset penalty term includes weighted summation of the squared parameter update offset of each parameter relative to the value obtained after training on the old task, using the diagonal elements of the Fisher information matrix of each parameter as weights; the new task classification loss term is the cross-entropy loss on the new task samples; the relative weight of the parameter update offset penalty term with respect to the new task classification loss term in the weighted combination is adjusted by the regularization constraint coefficient, and the regularization constraint coefficient is adaptively adjusted according to the classification accuracy loss of the old task on the validation set. When the classification accuracy loss of the old task exceeds a preset upper limit, the regularization constraint coefficient is increased by a preset step size; when the rate of decrease of the new task classification loss is lower than a preset lower limit, the regularization constraint coefficient is decreased by a preset step size.

[0012] Preferably, the method further includes: performing standardized preprocessing on the accessed multi-source heterogeneous data before the encoding; the standardized preprocessing includes: establishing a unified metadata dictionary, mapping the field types and field names in each data source to the standard field types and standard field names agreed upon by the unified metadata dictionary; performing fragmentation processing on the unstructured data in the multi-source heterogeneous data, and establishing an association index between the unstructured data blocks and the structured fields in the multi-source heterogeneous data.

[0013] Preferably, the method further includes: displaying the classification result and receiving manual correction of the classification result; recording the samples corresponding to the manually confirmed correction results as correction samples; when the cumulative number of recorded correction samples reaches a preset threshold, using the accumulated correction samples as new task data, triggering the classification model to perform new task training in accordance with the above-mentioned parameter update offset penalty term constraint to update the classification model.

[0014] Preferably, determining the business subject domain, sensitivity level, and field-level data lineage of the multi-source heterogeneous data using the classification result as an anchor point includes: mapping the cataloging category corresponding to the classification result to the corresponding business subject domain according to the preset correspondence between cataloging categories and business subject domains; marking the corresponding sensitivity level of the multi-source heterogeneous data based on the OR relationship determination between the classification result and the preset sensitive information regular expression pattern, wherein the OR relationship determination means that the sensitivity level marking is triggered when the classification result hits the preset sensitive category or the original content of the multi-source heterogeneous data matches any item in the preset sensitive information regular expression pattern; recording the entire process transformation path of the multi-source heterogeneous data from data source access, cross-modal feature fusion, data cataloging and classification to cataloging information entry, and constructing a field-level data lineage graph with fields as nodes and processing steps as edges in conjunction with task scheduling logs.

[0015] This invention also discloses an intelligent data cataloging system for heterogeneous data-guided cross-modal fusion and continuous learning, comprising: The data access module is configured to access multi-source heterogeneous data; The cross-modal feature fusion module is configured to encode the structured metadata, unstructured text data, and unstructured image data in the accessed multi-source heterogeneous data to obtain structured pattern vectors, text content vectors, and image visual vectors respectively; and to perform cross-modal attention fusion using the structured pattern vector as the query vector and the combined vector formed by concatenating the text content vector and the image visual vector along the feature dimension as the key vector and value vector respectively to obtain a fused semantic tensor. The incremental classification module is configured to take the fused semantic tensor as input, perform data cataloging and classification on the multi-source heterogeneous data through a classification model to obtain classification results; and when the classification model learns new data categories, it uses the diagonal elements of the Fisher information matrix of the loss function of the classification model with respect to each parameter after training on the old task as importance indicators, and weights the parameter update offset of each parameter according to the importance indicators to obtain a parameter update offset penalty term, and constructs the new task training objective as a weighted combination of the new task classification loss term and the parameter update offset penalty term; The multidimensional cataloging generation module is configured to use the classification results as anchors to determine the business subject domain, sensitivity level, and field-level data lineage of the multi-source heterogeneous data, thereby forming multidimensional cataloging information.

[0016] Preferably, the system further includes: a standardization preprocessing module, configured to perform standardization preprocessing on the accessed multi-source heterogeneous data before encoding by the cross-modal feature fusion module; the standardization preprocessing includes establishing a unified metadata dictionary and mapping the field types and field names in each data source to the standard field types and standard field names agreed upon in the unified metadata dictionary; and performing fragmentation processing on the unstructured data in the multi-source heterogeneous data and establishing an association index between the unstructured data blocks and the structured fields in the multi-source heterogeneous data; a human-computer interaction module, configured to display the classification results and receive manual corrections to the classification results; record the samples corresponding to the manually confirmed correction results as correction samples; and when the cumulative number of recorded correction samples reaches a preset threshold, trigger the incremental classification module to use the accumulated correction samples as new task data to train the classification model for a new task; and a cataloging module, configured to store the multi-source heterogeneous data processed by the standardization preprocessing module, the fused semantic tensor, the classification results, and the multidimensional cataloging information.

[0017] Preferably, the incremental classification module is further configured as follows: the parameter update offset penalty term obtained by weighting the parameter update offset of each parameter according to the importance index is implemented by weighting and summing the squares of the parameter update offset of each parameter relative to the value obtained after training on the old task, using the diagonal elements of the Fisher information matrix of each parameter as weights; the cross-entropy loss on the new task samples is used as the new task classification loss term; the relative weight of the parameter update offset penalty term relative to the new task classification loss term in the weighted combination is adjusted by the regularization constraint coefficient, and the regularization constraint coefficient is adaptively adjusted according to the classification accuracy loss of the old task on the validation set. When the classification accuracy loss of the old task exceeds the preset upper limit, the regularization constraint coefficient is increased by a preset step size; when the rate of decrease of the classification loss of the new task is lower than the preset lower limit, the regularization constraint coefficient is decreased by a preset step size.

[0018] Preferably, the multidimensional cataloging generation module is further configured to: map the cataloging category corresponding to the classification result to the corresponding business subject domain according to the preset correspondence between cataloging categories and business subject domains; mark the corresponding sensitivity level of the multi-source heterogeneous data based on the OR relationship determination between the classification result and the preset sensitive information regular expression pattern, wherein the OR relationship determination means that the sensitivity level marking is triggered when the classification result hits the preset sensitive category or the original content of the multi-source heterogeneous data matches any item in the preset sensitive information regular expression pattern; record the entire process transformation path of the multi-source heterogeneous data from data source access to cataloging information entry, and construct a field-level data lineage graph with fields as nodes and processing steps as edges in combination with task scheduling logs.

[0019] Beneficial effects 1. Cross-modal attention fusion is performed using structured pattern vectors as query vectors and combined vectors formed by concatenating unstructured text and unstructured images along feature dimensions as key vectors and value vectors, respectively. The semantic tensor output by fusion carries both structured semantic constraints and unstructured local details. Compared with existing solutions that process structured and unstructured data separately or symmetrically concatenate features on both sides, this invention focuses on unstructured local content by structured metadata during the fusion stage, thereby improving the accuracy of cross-modal semantic alignment.

[0020] 2. By using the importance index of parameters to the classification performance of old tasks as weights to apply a weighted penalty to the square of parameter update offset, the classification model can automatically determine the degree of freedom to update according to the importance of parameters when learning new data categories. Parameters that are important to the old tasks are strictly protected, while parameters that have little impact on the old tasks can be updated freely. Compared with the existing approach of retraining the model with all data or making simple fine-tuning, this invention can maintain the classification performance of old categories in engineering scenarios where new data categories are continuously introduced.

[0021] 3. Using the classification results as a unified anchor point, the three cataloging dimensions of business subject domain, sensitivity level, and field-level data lineage are derived synchronously. This ensures that the three cataloging dimensions are generated based on the same classification input, avoiding inconsistencies when the three cataloging dimensions are generated independently. It also automates the cataloging dimensions that originally relied on manual completion. Attached Figure Description

[0022] Figure 1 This is a flowchart illustrating a preferred embodiment of the intelligent data cataloging method for heterogeneous data-guided cross-modal fusion and continuous learning provided by the present invention. Figure 2 This is a schematic diagram of the structure of an intelligent data cataloging system for heterogeneous data-guided cross-modal fusion and continuous learning provided in a preferred embodiment of the present invention; Figure 3 This is a schematic diagram of a cross-modal feature fusion process provided in a preferred embodiment of the present invention. Detailed Implementation

[0023] To make the objectives, technical solutions, and beneficial effects of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. The embodiments described are only for explaining the invention and are not intended to limit the scope of protection of the invention.

[0024] Example 1 like Figure 1As shown, this embodiment provides an intelligent data cataloging method for cross-modal fusion and continuous learning guided by heterogeneous data. It includes four core steps, S1 to S4, and in some preferred embodiments, also includes an optional preparatory step S0 and an optional extension step S5. This embodiment uses a large manufacturing enterprise as a deployment scenario. This enterprise has approximately 20 heterogeneous data sources, covering four types of data: relational databases, PDF / Word business documents, production monitoring images, and message queue data, with approximately 120 million records to be cataloged. Those skilled in the art should understand that the applicability of this method is not limited to the number of data sources and the scale of data in this scenario.

[0025] Step S1: Encode the structured metadata, unstructured text data, and unstructured image data in the multi-source heterogeneous data to obtain structured pattern vectors, text content vectors, and image visual vectors respectively.

[0026] This step encodes the three heterogeneous data streams into comparable semantic vectors within the same vector space, providing input for subsequent cross-modal attention fusion.

[0027] Encode structured metadata (including table structure, field names, and field comments) and output a structured schema vector. In one specific embodiment, a pre-trained language model is used to encode the field names, field comments, and table structure information in the structured metadata by concatenating them into short text according to a preset template. The pre-trained language model used is the BERT-base model. The overall semantic representation vector output by the pre-trained language model after encoding the short text is taken. In engineering implementation, the preset template can be expressed as a fixed concatenation of "[field name]|[field comment]|belonging table:[table structure information]", and the concatenated short text serves as the input sequence for the pre-trained language model. The pre-trained language model can also use other language models with equivalent encoding capabilities, such as RoBERTa, ALBERT, and ERNIE; the specific type of the pre-trained language model is not limited.

[0028] Unstructured text data (including document text, OCR recognition results, production log text, etc.) is encoded to output text content vectors. To ensure and Comparable within the same semantic space, the encoding of unstructured text data and the encoding of structured metadata employ a pre-trained language model with the same origin. This "same origin" means that both share a basic vocabulary and are pre-trained on the same pre-training corpus, with parameters that can be shared or independently fine-tuned. In one specific embodiment, both use the BERT-base model, and the encoding method is consistent with the aforementioned encoding method for structured metadata.

[0029] Encode unstructured image data to output image visual vectors. In one specific embodiment, a convolutional neural network is used to extract features from image slices; the convolutional neural network used is the ResNet50 model. Take the output vector of the last global average pooling layer of the convolutional neural network. The convolutional neural network described above can also be other convolutional neural networks such as VGG, DenseNet, and EfficientNet, or a Vision Transformer-type visual coding network can be used to encode the image; the specific type of image coding network is not limited.

[0030] The three encoding paths described above process structured metadata, unstructured text data, and unstructured image data, respectively. The three output vectors can be unified to the same feature dimension by the projection layers at the ends of each encoding network, facilitating subsequent attention calculations. In other embodiments, if the text data and image data come from the same document (e.g., a PDF business document with images), the document pagination granularity can be optimized. and Establish a homogeneity identifier for optional alignment enhancement in subsequent fusion stages.

[0031] Step S2: Using the structured pattern vector as the query vector, and the combined vector formed by concatenating the text content vector and the image visual vector along the feature dimension as the key vector and value vector respectively, cross-modal attention fusion is performed to obtain the fused semantic tensor.

[0032] like Figure 3 As shown, this step will transform the structured pattern vector. Text content vector With image visual vectors Feed it into the cross-modal attention mechanism. First, and Concatenate along the feature dimension to form a combined vector; The query vector, after being projected, serves as the query vector. The combined vector, after being projected by the key and value, serves as the key vector and value vector, respectively. A scaling dot product attention operation is performed on the query, key, and value to obtain the fused semantic tensor. .

[0033] In one specific embodiment, the construction method of the query, key, and value is determined by the following formula: ; in, Represents the query vector; Represents the key vector; The value vector in the attention mechanism is represented by the symbol of the three feature vectors. , , The presence or absence of a subscript distinguishes between them, thus avoiding symbolic conflict; This indicates a query for the projection weight matrix; This represents the key projection weight matrix; The values ​​represent the projected weight matrix, and all three are learnable parameters of the model. This represents a combined vector obtained by concatenating the text content vector and the image visual vector along the feature dimension.

[0034] In one specific embodiment, the fusion semantic tensor is determined by the following formula: ; in, Represents a fusion semantic tensor; , , Same meaning as before; To represent the feature dimension of the key vector, we introduce... Scale the dot product result to avoid excessively large dot product values. Output tends to be hard-assigned; This represents a row-normalized exponential function that operates on each row of a scaled dot product matrix, such that the sum of the weights of each query position with respect to all key positions is 1.

[0035] In other embodiments, the cross-modal attention can take a multi-head form: , , The code segments the data into several heads along the feature dimension, calculates attention for each head separately, and then concatenates them to ensure the fused output captures semantic correspondences at multiple granularities. Alternatively, it can be... After output, residuals and layer normalization are introduced to stabilize training.

[0036] Step S3: Using the fused semantic tensor as input, the multi-source heterogeneous data is cataloged and classified through the classification model to obtain the classification result; and when the classification model learns new data categories, the diagonal elements of the Fisher information matrix of the loss function of the classification model with respect to each parameter after the old task training are used as the importance index of each parameter to the classification performance of the old task, and the training objective of the new task is constructed as a weighted combination of the new task classification loss term and the parameter update offset penalty term.

[0037] This step is divided into two stages on the classification side: the basic classification stage directly uses the fused semantic tensor. As input, the probability distribution of the cataloging category to which the current data belongs is output, and the classification result is given accordingly. In the incremental learning stage, when the classification model has completed the training of the old task and needs to learn the new data category, the parameter update offset is subject to regularization constraint according to the importance index of the classification performance of the old task.

[0038] Classification side: A fully connected neural network is used as the base classification network to fuse semantic tensors. As input, it passes through several fully connected layers and activation layers before being processed by... The output layer outputs the probability distribution of the data across each cataloging category, and the category with the highest probability is taken as the classification result. In one specific embodiment, the basic classification network consists of two fully connected layers and one output layer, with ReLU as the activation function and cross-entropy loss as the classification loss. In other embodiments, the classification network may also employ residual fully connected structures, attention-based classification heads, or other structures; the specific structure of the classification network is not limited.

[0039] On the incremental side: Scenarios such as "adding a new data source," "adding a new business data category," and "updating the data format" are defined as new incremental learning tasks. The samples and labels in the new task training set are denoted as the new task data. Before training the new task, the classification model is first trained on the old task training set. The parameters of the classification model when the old task training is completed are denoted as... This is used as a reference point for parameter offset. In constructing the training objective for the new task, the importance of the parameters to the classification performance of the old task is used as the weight, and the parameters are weighted relative to... The squared parameter update offsets are weighted and summed to form a parameter update offset penalty term. This penalty term is then weighted and combined with the new task classification loss term to obtain the new task training objective. Regularization constraint coefficients adjust the relative weight of the penalty term with respect to the new task classification loss term. During the new task training phase, the classification model updates its parameters according to the new task training objective. Parameters important to the old task are strictly protected, while parameters with less impact on the old task are allowed larger parameter update offsets.

[0040] Specific implementation of the parameter importance index: In one specific embodiment, the importance index of the parameter to the classification performance of the old task is taken as the diagonal element of the Fisher information matrix of the loss function of the classification model with respect to the parameter after the old task training is completed. (Fisher information matrix diagonal elements) We measure the second-order curvature of the loss function at this parameter. A larger curvature means that a small perturbation to this parameter will cause a drastic change in the loss, thus contributing more to the classification performance of the old task. A smaller curvature means that the local loss function of this parameter is not sensitive to its perturbation, and the parameter has less impact on the old task and can be freely updated on the new task. Using this curvature as the importance weight of this parameter allows the training of the new task to impose stronger constraints on parameters with high curvature and retain more degrees of freedom for updating parameters with low curvature, thus forming a causal chain of "curvature → importance → constraint strength" at the mathematical level.

[0041] In one specific embodiment, the new task training objective is determined by the following formula: ; in, This represents the set of all learnable parameter vectors for the classification model, and the parameters updated during the training phase for a new task are... ; express The Middle Components, index Iterate through all learnable parameters of the classification model; Indicates the completion of the training for the old task. The values ​​of each parameter are used as reference points for parameter offset; The diagonal of the Fisher information matrix represents the first... The element represents the element. The importance of these parameters to the classification performance of old tasks; This represents the regularization constraint coefficient, used to adjust the relative weight of the parameter update offset penalty term with respect to the new task classification loss term; This represents the new task classification loss term, which in a specific embodiment is taken as the cross-entropy loss; The training objective for the new task is represented by a weighted combination of the new task classification loss term and the parameter update offset penalty term; the summation term... That is, the parameter updates the offset penalty term.

[0042] Regularization constraint coefficient The principle for determining the value: The larger the value, the higher the relative weight of the parameter update offset penalty term in the training objective, the stronger the protection of important parameters of the old task, and the lower the tolerance for learning of the new task; The smaller the value, the higher the tolerance for learning new tasks, and the weaker the protection of important parameters of old tasks. In one specific embodiment, take... In actual deployment, the value can be adjusted within the range of [50, 500] according to the relative importance of the new task and the old task. When the difference between the new data category and the existing category is large and the new task is expected to learn more fully, the value should be taken near the lower limit. When the business side has high requirements for the stability of the classification performance of the old category and wants to prioritize the protection of the knowledge of the old task, the value should be taken near the upper limit.

[0043] In some preferred embodiments, the regularization constraint coefficient can also be adaptively adjusted based on the classification accuracy loss of the old task on the validation set. If the old task accuracy loss on the validation set exceeds a preset upper limit, the regularization constraint coefficient is increased by a preset step size; if the rate of decrease in the classification loss of the new task on the validation set is lower than a preset lower limit, the regularization constraint coefficient is decreased by a preset step size. The adjustment frequency can be consistent with the training epoch. In a specific embodiment, the preset upper limit is 5% of the old task classification accuracy loss, the preset lower limit is 1% of the rate of decrease in the new task classification loss per epoch, and the preset step size is the current step size. The value is 10%. The above adaptive adjustment mechanism enables... The value can be dynamically adjusted based on two-way feedback during training, avoiding the limitation of a single fixed value on the ability to balance new and old tasks. The specific value and adjustable range are not limited.

[0044] In other embodiments, the importance index of the parameters to the classification performance of old tasks can also be other metrics such as parameter path integral estimated by synaptic intelligence or output sensitivity estimated by memory perception.

[0045] Step S4: Using the classification results as anchors, determine the business subject domain, sensitivity level, and field-level data lineage of the multi-source heterogeneous data to form multidimensional cataloging information.

[0046] This step uses the classification results output in step S3 as a unified anchor point to derive cataloging information in parallel in three directions, and then converges it into unified multidimensional cataloging information at the output end.

[0047] On the subject domain mapping side: a pre-established correspondence between cataloging categories and business subject domains is used; during inference, the cataloging category corresponding to the classification result is mapped to the corresponding business subject domain according to this correspondence. In a specific embodiment, the correspondence is pre-configured in ways such as "Customer Information," "Customer Domain," "Financial Data," "Financial Domain," "Production Data," and "Production Domain"; when a piece of data is classified as "Customer Information," it is automatically attached to the "Customer Domain." The specific content of the correspondence is configured according to the enterprise's business needs, and there is no limitation on the content of the specific mapping table.

[0048] Sensitivity Level Identification Side: Sensitivity level is determined and marked based on the OR relationship between the classification result and a preset sensitive information regular expression pattern. The OR relationship determination means that a sensitivity level marking is triggered when the classification result matches a preset sensitive category or when the original content of the multi-source heterogeneous data matches any item in the preset sensitive information regular expression pattern. In one specific embodiment, the preset sensitive information regular expression pattern includes regular expressions for ID card numbers (18 digits), mobile phone numbers (11 digits), and bank card numbers (16-19 digits). When a piece of data is classified into categories such as "personal privacy" or "financially sensitive," or when its original content matches any item in the above sensitive information regular expression pattern, it is marked according to the corresponding level in L1-normal, L2-relatively sensitive, and L3-sensitive classification. The two input paths of the OR relationship determination (classification result matching a preset sensitive category and original content matching a regular expression pattern) are independent of each other; a match in either path triggers the marking. When both paths match simultaneously, the higher sensitivity level in the classification is taken. The classification granularity can be expanded to four or more levels according to compliance requirements.

[0049] On the field-level lineage construction side: This records the entire transformation path of the multi-source heterogeneous data, from data source access, cross-modal feature fusion, data cataloging and classification to cataloging information entry into the database. It also constructs a field-level data lineage graph with fields as nodes and processing steps as edges, based on task scheduling logs. In one specific embodiment, the field-level data lineage graph is stored in a graph database. Node attributes record the data source, field name, and field type of the field, while edge attributes record the type of processing steps and timestamps experienced by the field. When a downstream user needs to trace the source of a cataloging record, they can traverse the lineage graph backwards to the data source; when a field on the data source side changes, they can traverse the lineage graph forwards to all affected cataloging records.

[0050] In some preferred embodiments, the method further includes step S0 before the encoding: standardizing and preprocessing the accessed multi-source heterogeneous data.

[0051] Step S0 receives raw data from various types of data sources and outputs preprocessed data with a consistent format and naming convention, and with an established association index between unstructured data blocks and structured fields, providing a consistent input basis for subsequent encoding and cross-modal fusion.

[0052] On the access side: Unified data collection is achieved using interface protocols adapted to various data sources, including relational databases, file servers, message queues, and image storage systems. In one specific embodiment, the relational database side interfaces with Oracle and MySQL to collect structured data such as order tables and employee information tables, respectively; the file server side collects unstructured text data such as business contracts and technical documents in PDF and Word formats; the image storage system side collects production monitoring images in JPG format; and the message queue side interfaces with Kafka to collect production log data. The above data source types and collection interfaces are merely examples; the same approach can be used when connecting to other types of data sources (such as object storage, time-series databases, HDFS, etc.), and there are no limitations on this.

[0053] On the cleaning side: The incoming raw data undergoes cleaning processes including deduplication, missing value imputation, outlier removal, and noise filtering. In one specific embodiment, missing values ​​in numeric fields are imputed using the mean of the same field across historical samples, while missing values ​​in character fields are marked as "unknown." Outlier removal employs the 3σ principle. In other embodiments, missing value imputation can utilize median, mode, regression imputation, or other statistical methods, and outlier identification can employ IQR or other statistical methods, without affecting the execution of subsequent steps.

[0054] On the dictionary side: A unified metadata dictionary is established, mapping the field types and names from various data sources to the standard field types and names agreed upon in the unified metadata dictionary. In one specific embodiment, Oracle's NUMBER type and MySQL's INT type are uniformly mapped to the INTEGER standard type; fields with the same semantics but different names from different data sources (e.g., "cust_id", "customer number", "customer ID") are uniformly mapped to the same standard field name "customer number". The unified metadata dictionary can be implemented as a relational table, a configuration file, or a graph database node; the storage medium is not limited.

[0055] On the indexing side: Unstructured data in multi-source heterogeneous data is segmented, and an association index is established between unstructured data blocks and structured fields in the multi-source heterogeneous data. In a specific embodiment, business documents are segmented by paragraph or fixed character length, and paragraph keywords are extracted as coarse-grained features during segmentation; surveillance images are segmented by region, and image texture features are extracted as coarse-grained features during segmentation; the aforementioned unstructured data blocks establish potential association indexes based on the semantic similarity between coarse-grained features and table rows and fields. For example, if the semantic vector similarity between the coarse-grained features of a surveillance image segment and the "customer number" field is higher than a preset threshold, then an edge between the image segment and the "customer number" field is established in the association index. The association index provides candidate alignment ranges for subsequent cross-modal fusion, avoiding indiscriminate matching on the entire unstructured data during the fusion stage.

[0056] In some preferred embodiments, the method further includes step S5: displaying the classification result and receiving manual correction of the classification result, recording the samples corresponding to the manually confirmed correction results as correction samples; when the cumulative number of recorded correction samples reaches a preset threshold, using the accumulated correction samples as new task data, triggering the classification model to perform new task training in accordance with the above parameter update offset penalty term constraint to update the classification model.

[0057] In one specific embodiment, the preset threshold is set to 50; in other embodiments, the preset threshold can be adjusted between 20 and 200 based on the business scale. When the business scale is small and the correction frequency is low, the threshold is taken near the lower limit; when the business scale is large and the correction samples accumulate rapidly, the threshold is taken near the upper limit, so that the triggering frequency matches the training cost. Step S5 connects the human-machine closed-loop feedback with the construction method of the incremental training objective in step S3. While manually correcting samples enters the model iteration at low cost, the parameter update offset penalty term ensures that the knowledge of the old task is not destroyed.

[0058] Example 2 like Figure 2 As shown, this embodiment provides an intelligent data cataloging system for heterogeneous data-guided cross-modal fusion and continuous learning, used to implement the intelligent data cataloging method described in Embodiment 1. The system includes four core modules: a data access module, a cross-modal feature fusion module, an incremental classification module, and a multi-dimensional cataloging generation module. In some preferred embodiments, the system also includes a standardization preprocessing module, a human-computer interaction module, and a cataloging library module. The modules are sequentially connected according to the data flow direction, with the output of one module serving as the input of the next.

[0059] The data access module is configured to access multi-source heterogeneous data. In one specific embodiment, this module adopts an access layer composed of interface protocols that adapt to various types of data sources such as relational databases, file servers, message queues, and image storage systems, exposing a unified collection API to the outside world; internally, it writes the collection results of each data source into the input buffer of the downstream module in the form of a triple of data source identifier, collection timestamp, and original data payload.

[0060] The cross-modal feature fusion module is configured to encode the structured metadata, unstructured text data, and unstructured image data from the accessed multi-source heterogeneous data, respectively, to obtain structured pattern vectors, text content vectors, and image visual vectors. It then uses the structured pattern vector as the query vector and the combined vector formed by concatenating the text content vector and the image visual vector along the feature dimension as the key vector and value vector, respectively, to perform cross-modal attention fusion, resulting in a fused semantic tensor. This module internally consists of a three-way encoding submodule and a cross-modal attention submodule. The three-way encoding submodule, as described in step S1 of embodiment one, uses a pre-trained language model and a convolutional neural network to encode the three types of data respectively. The cross-modal attention submodule, as described in step S2 of embodiment one... - - The construction method and the scaling dot product attention formula are used to calculate the fused semantic tensor.

[0061] The incremental classification module is configured to take the fused semantic tensor as input, perform data cataloging and classification on the multi-source heterogeneous data through a classification model to obtain classification results; and when the classification model learns new data categories, it uses the diagonal elements of the Fisher information matrix of each parameter of the loss function of the classification model after training on the old task as the importance index of each parameter to the classification performance of the old task. The training objective of the new task is constructed as a weighted combination of the new task classification loss term and the parameter update offset penalty term. The parameter update offset penalty term is weighted and summed by the square of the parameter update offset of each parameter relative to the value obtained after training on the old task, using the diagonal elements of the Fisher information matrix of each parameter as weights, and the relative weight of the parameter update offset penalty term to the new task classification loss term is adjusted by a regularization constraint coefficient. This module consists of three parts: a basic classification submodule, a parameter importance evaluation submodule, and an incremental training submodule; the basic classification submodule undertakes the forward inference and old task training of the fully connected classification network described in step S3 of embodiment one; the parameter importance evaluation submodule calculates the importance of the new task classification loss term after training on the old task. Incremental training submodule according to Conduct training for new tasks.

[0062] In one specific embodiment, the incremental classification module is further configured to: use the cross-entropy loss on the new task samples as the classification loss term for the new task; and adaptively adjust the regularization constraint coefficient according to the classification accuracy loss of the old task on the validation set. When the classification accuracy loss of the old task exceeds a preset upper limit, the regularization constraint coefficient is increased by a preset step size; when the rate of decrease of the classification loss of the new task is lower than a preset lower limit, the regularization constraint coefficient is decreased by a preset step size. This adaptive adjustment can be implemented in engineering by a scheduling logic shared by the parameter importance evaluation submodule and the incremental training submodule, reading validation set feedback and updating it according to the training epoch. .

[0063] The multidimensional cataloging generation module is configured to use the classification results as anchors to determine the business subject domain, sensitivity level, and field-level data lineage of the multi-source heterogeneous data, forming multidimensional cataloging information. This module consists of three parts: a subject domain mapping submodule, a sensitivity level identification submodule, and a field-level lineage construction submodule. Following the correspondence mapping, OR relationship determination, and full-process conversion path recording methods described in step S4 of Embodiment 1, it undertakes the derivation of the three cataloging dimensions respectively, and integrates the three dimensions with the classification results into unified multidimensional cataloging information at the output end.

[0064] In one specific embodiment, the multidimensional cataloging generation module is further configured to: map the cataloging category corresponding to the classification result to the corresponding business subject domain according to the preset correspondence between cataloging categories and business subject domains; mark the corresponding sensitivity level of the multi-source heterogeneous data based on the OR relationship determination between the classification result and the preset sensitive information regular expression pattern, wherein the OR relationship determination means that the sensitivity level marking is triggered when the classification result hits the preset sensitive category or the original content of the multi-source heterogeneous data matches any item in the preset sensitive information regular expression pattern; record the entire process transformation path of the multi-source heterogeneous data from data source access to cataloging information entry, and construct a field-level data lineage graph with fields as nodes and processing steps as edges in combination with task scheduling logs.

[0065] In some preferred embodiments, the system further includes three optional modules: a standardization preprocessing module, a human-computer interaction module, and a cataloging library module.

[0066] The standardization preprocessing module is configured to perform standardization preprocessing on the incoming multi-source heterogeneous data before encoding by the cross-modal feature fusion module. The internal processing of this module is consistent with step S0 in Embodiment 1, comprising three parts: a cleaning submodule, a unified metadata dictionary submodule, and an association index submodule. The cleaning submodule performs deduplication, missing value imputation, and outlier removal. The unified metadata dictionary submodule maps the field types and field names of each data source to a unified standard. The association index submodule segments the unstructured data and establishes association indexes between the unstructured data and the structured fields.

[0067] The human-computer interaction module is configured to display the classification results and receive manual corrections to the classification results. Samples corresponding to manually confirmed corrections are recorded as correction samples. When the cumulative number of recorded correction samples reaches a preset threshold, the incremental classification module is triggered to use the accumulated correction samples as new task data to train the classification model for a new task. In one specific embodiment, the human-computer interaction module displays the classification results and corresponding original data fragments in a visual interface. Maintenance personnel review each sample and correct any incorrectly classified samples. The preset threshold is 50 samples, which can be adjusted from 20 to 200 samples depending on the scale of the business.

[0068] The cataloging repository module is configured to store the multi-source heterogeneous data processed by the standardization preprocessing module, the fused semantic tensor, the classification results, and the multidimensional cataloging information. In one specific embodiment, the cataloging repository module uses a combination of a relational database and an index database for storage. The structured data and multidimensional cataloging information after standardization preprocessing are stored in the relational database (e.g., PostgreSQL), while the fused semantic tensor and unstructured content index are stored in the index database (e.g., Elasticsearch). The two are linked through the primary key of the cataloging record. The specific storage medium for the cataloging repository can be selected according to the deployment environment and is not limited thereto.

[0069] Experimental Example To verify the feasibility, practicality, and technical advantages of the present invention compared to existing solutions, it was deployed and tested in an enterprise production environment according to Embodiment 1 and Embodiment 2.

[0070] Deployment environment: Hardware uses an Intel Xeon Gold 6248 (40 cores) CPU, 256 GB of memory, an NVIDIA Tesla V100 32 GB GPU, and a 20 TB NVMe SSD; the operating system is CentOS 8; the database uses a combination of PostgreSQL 13 and Elasticsearch 7.10; the algorithm framework uses TensorFlow 2.5 for BERT and ResNet50 model training and inference, distributed data processing uses Apache Spark 3.0; incremental learning is implemented using PyTorchLightning.

[0071] Datasets and Scenarios: The tests are based on a private dataset of approximately 120 million heterogeneous data points from multiple sources, covering structured, semi-structured, and unstructured data, supplemented by two public datasets: DBpedia and IMDb. The test scenarios focus on three aspects: cross-modal semantic alignment capability, incremental learning resistance to forgetting, and automated multi-dimensional cataloging capability. The control group includes traditional rule-based methods, SVM methods, and deep learning methods without fused features.

[0072] Parameter configuration: The pre-trained language model uses BERT-base, and the convolutional neural network uses ResNet50; regularization constraint coefficients... Set the sample size to 100; adjust the sample threshold to 50; the sensitivity level regularization mode is preset to three categories: ID card number (18 digits), mobile phone number (11 digits), and bank card number (16-19 digits).

[0073] Test Results: The test results are shown in Table 1 below. Under the aforementioned deployment environment, the cataloging accuracy of the method of this invention reached 96.8%, the cross-modal alignment accuracy reached 93.2%, the convergence time for new categories (represented by "production equipment failure images") was 0.8 hours, the accuracy decrease rate for old categories was 3.2%, and the manual intervention rate was 3.8%. Under the same caliber, the cataloging accuracy of the traditional rule method was 78.5%, and it did not support learning new categories; the cataloging accuracy of the SVM method was 85.2%, the convergence time for new categories was 2.5 hours, the accuracy decrease rate for old categories was 8.3%, and the manual intervention rate was 32.1%; the cataloging accuracy of the deep learning method without fused features was 88.3%, the convergence time for new categories was 1.6 hours, the accuracy decrease rate for old categories was 6.7%, and the manual intervention rate was 18.5%. In a real-world deployment at a large manufacturing enterprise, the cataloging accuracy of the method of this invention was 96.7%, the F1 score was 0.960, the processing time for a single data entry was 19.3 ms, the adaptive convergence time for the newly added "production equipment fault images" category was 0.75 hours, the accuracy decline rate of the old category was 3.1%, the manual intervention rate was 3.7%, the pedigree graph coverage was 100%, and the field-level pedigree tracing accuracy was 98.7%.

[0074] Table 1 Test Results In the above results: In terms of cross-modal semantic alignment, the matching accuracy between the "amount" field and the OCR digital area reached 93.2%, verifying the ability of cross-modal attention fusion, which uses structured pattern vectors as query vectors, to establish a correspondence between structured fields and unstructured local content. In terms of incremental learning to resist forgetting, after adding the "production equipment failure image" category, the accuracy of the old category decreased by 3.2% (3.1% in actual deployment), which is significantly lower than the 8.3% of the SVM method and the 6.7% of the non-fusion deep learning method, verifying the ability of constrained incremental learning, represented by the parameter update offset penalty term, to protect the knowledge of the old task. In terms of automated cataloging, the automatic generation rate of multidimensional cataloging information reached 95%, and the manual intervention rate dropped to 3.8% (3.7% in actual deployment). This is a significant decrease compared to 18.5% for methods without deep learning integration and 32.1% for SVM methods, verifying the automated benefits of determining the three cataloging dimensions using classification results as anchors.

[0075] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the present invention as claimed. The scope of protection of this invention is defined by the appended claims and their equivalents.

Claims

1. A smart data cataloging method guided by heterogeneous data for cross-modal fusion and continuous learning, characterized in that, include: The structured metadata, unstructured text data, and unstructured image data in the multi-source heterogeneous data are encoded respectively to obtain structured pattern vectors, text content vectors, and image visual vectors. Using the structured pattern vector as the query vector, and the combined vector formed by concatenating the text content vector and the image visual vector along the feature dimension as the key vector and value vector respectively, cross-modal attention fusion is performed to obtain the fused semantic tensor; Using the fused semantic tensor as input, the multi-source heterogeneous data is cataloged and classified through a classification model to obtain the classification result; When the classification model learns new data categories, the diagonal elements of the Fisher information matrix of the loss function of the classification model with respect to each parameter after the old task training are used as importance indicators. The parameter update offset of each parameter is weighted according to the importance indicators to obtain the parameter update offset penalty term. The training objective of the new task is constructed as a weighted combination of the new task classification loss term and the parameter update offset penalty term. Using the classification results as anchors, the business subject domain, sensitivity level, and field-level data lineage of the multi-source heterogeneous data are determined to form multi-dimensional cataloging information.

2. The intelligent data cataloging method for heterogeneous data-guided cross-modal fusion and continuous learning according to claim 1, characterized in that, The encoding of the structured metadata and the encoding of the unstructured text data are achieved through a pre-trained language model with the same origin, where the same origin means that the two share a basic vocabulary and are pre-trained based on the same pre-training corpus. The structured metadata is encoded by concatenating field names, field comments, and the structure information of the table to which they belong using a preset template, and then inputting the short text into the pre-trained language model of the same origin; the encoding of the unstructured image data is achieved through a convolutional neural network.

3. The intelligent data cataloging method for heterogeneous data-guided cross-modal fusion and continuous learning according to claim 1, characterized in that, The step of weighting the parameter update offset of each parameter according to the importance index to obtain the parameter update offset penalty term includes weighted summation of the squared parameter update offset of each parameter relative to the value obtained after training on the old task, using the diagonal elements of the Fisher information matrix of each parameter as weights; the new task classification loss term is the cross-entropy loss on the new task samples; the relative weight of the parameter update offset penalty term with respect to the new task classification loss term in the weighted combination is adjusted by the regularization constraint coefficient, and the regularization constraint coefficient is adaptively adjusted according to the classification accuracy loss of the old task on the validation set. When the classification accuracy loss of the old task exceeds the preset upper limit, the regularization constraint coefficient is increased by a preset step size; when the rate of decrease of the new task classification loss is lower than the preset lower limit, the regularization constraint coefficient is decreased by a preset step size.

4. The intelligent data cataloging method for heterogeneous data-guided cross-modal fusion and continuous learning according to claim 1, characterized in that, The method further includes: performing standardized preprocessing on the accessed multi-source heterogeneous data before encoding; the standardized preprocessing includes: establishing a unified metadata dictionary, mapping the field types and field names in each data source to the standard field types and standard field names agreed upon by the unified metadata dictionary; performing fragmentation processing on the unstructured data in the multi-source heterogeneous data, and establishing an association index between the unstructured data blocks and the structured fields in the multi-source heterogeneous data.

5. The intelligent data cataloging method for heterogeneous data-guided cross-modal fusion and continuous learning according to claim 1, characterized in that, The method further includes: displaying the classification result and receiving manual correction of the classification result; recording the samples corresponding to the manually confirmed correction results as correction samples; when the cumulative number of recorded correction samples reaches a preset threshold, using the accumulated correction samples as new task data, triggering the classification model to perform new task training in the manner described in claim 1 to update the classification model.

6. The intelligent data cataloging method for heterogeneous data-guided cross-modal fusion and continuous learning according to claim 1, characterized in that, Using the classification results as anchors, the business subject domain, sensitivity level, and field-level data lineage of the multi-source heterogeneous data are determined, including: mapping the cataloging category corresponding to the classification result to the corresponding business subject domain according to the preset correspondence between cataloging categories and business subject domains; marking the corresponding sensitivity level of the multi-source heterogeneous data based on the OR relationship determination between the classification result and the preset sensitive information regular expression pattern, wherein the OR relationship determination means that the sensitivity level marking is triggered when the classification result hits the preset sensitive category or the original content of the multi-source heterogeneous data matches any item in the preset sensitive information regular expression pattern; recording the entire process transformation path of the multi-source heterogeneous data from data source access, cross-modal feature fusion, data cataloging and classification to cataloging information entry, and constructing a field-level data lineage graph with fields as nodes and processing steps as edges in combination with task scheduling logs.

7. An intelligent data cataloging system for heterogeneous data-guided cross-modal fusion and continuous learning, characterized in that, include: The data access module is configured to access multi-source heterogeneous data; The cross-modal feature fusion module is configured to encode the structured metadata, unstructured text data, and unstructured image data in the accessed multi-source heterogeneous data to obtain structured pattern vectors, text content vectors, and image visual vectors respectively; and to perform cross-modal attention fusion using the structured pattern vector as the query vector and the combined vector formed by concatenating the text content vector and the image visual vector along the feature dimension as the key vector and value vector respectively to obtain a fused semantic tensor. The incremental classification module is configured to take the fused semantic tensor as input and perform data cataloging and classification on the multi-source heterogeneous data through a classification model to obtain the classification result. When the classification model learns new data categories, the diagonal elements of the Fisher information matrix of the loss function of the classification model with respect to each parameter after the old task training is completed are used as importance indicators. The parameter update offset of each parameter is weighted according to the importance indicators to obtain the parameter update offset penalty term. The training objective of the new task is constructed as a weighted combination of the new task classification loss term and the parameter update offset penalty term. The multidimensional cataloging generation module is configured to use the classification results as anchors to determine the business subject domain, sensitivity level, and field-level data lineage of the multi-source heterogeneous data, thereby forming multidimensional cataloging information.

8. The intelligent data cataloging system for heterogeneous data-guided cross-modal fusion and continuous learning according to claim 7, characterized in that, The system also includes: The standardization preprocessing module is configured to perform standardization preprocessing on the accessed multi-source heterogeneous data before the cross-modal feature fusion module performs encoding. The standardization preprocessing includes establishing a unified metadata dictionary and mapping the field types and field names in each data source to the standard field types and standard field names agreed upon in the unified metadata dictionary, as well as performing fragmentation processing on the unstructured data in the multi-source heterogeneous data and establishing an association index between the unstructured data blocks and the structured fields in the multi-source heterogeneous data. The human-computer interaction module is configured to display the classification results and receive manual corrections to the classification results. The module records the samples corresponding to the manually confirmed correction results as correction samples. When the cumulative number of the recorded correction samples reaches a preset threshold, the module triggers the incremental classification module to use the accumulated correction samples as new task data to train the classification model for a new task. The cataloging module is configured to store the multi-source heterogeneous data processed by the standardization preprocessing module, the fused semantic tensor, the classification results, and the multidimensional cataloging information.

9. The intelligent data cataloging system for heterogeneous data-guided cross-modal fusion and continuous learning according to claim 7, characterized in that, The incremental classification module is further configured as follows: the parameter update offset penalty term obtained by weighting the parameter update offset of each parameter according to the importance index is implemented by weighting and summing the squares of the parameter update offset of each parameter relative to the value obtained after training on the old task, using the diagonal elements of the Fisher information matrix of each parameter as weights; the cross-entropy loss on the new task samples is used as the new task classification loss term; the relative weight of the parameter update offset penalty term relative to the new task classification loss term in the weighted combination is adjusted by the regularization constraint coefficient, and the regularization constraint coefficient is adaptively adjusted according to the classification accuracy loss of the old task on the validation set. When the classification accuracy loss of the old task exceeds the preset upper limit, the regularization constraint coefficient is increased by a preset step size; when the rate of decrease of the classification loss of the new task is lower than the preset lower limit, the regularization constraint coefficient is decreased by a preset step size.

10. The intelligent data cataloging system for heterogeneous data-guided cross-modal fusion and continuous learning according to claim 7, characterized in that, The multidimensional cataloging generation module is further configured to: map the cataloging category corresponding to the classification result to the corresponding business subject domain according to the preset correspondence between cataloging categories and business subject domains; Based on the OR relationship determination between the classification result and the preset sensitive information regularity pattern, the corresponding sensitivity level is marked for the multi-source heterogeneous data. The OR relationship determination means that the sensitivity level marking is triggered when the classification result hits the preset sensitive category or the original content of the multi-source heterogeneous data matches any item in the preset sensitive information regularity pattern. The entire transformation path of the multi-source heterogeneous data from data source access to catalog information entry is recorded, and a field-level data lineage graph with fields as nodes and processing steps as edges is constructed by combining task scheduling logs.

Images