An artificial intelligence-based data cataloging method, system, device and storage medium

Abstract

The application relates to the technical field of data processing, and relates to a data cataloging method, system and device based on artificial intelligence and a storage medium. The data cataloging method based on artificial intelligence comprises the following steps: obtaining business data to be processed; performing modal recognition and keyword extraction on the business data to obtain a first classification result and a data type; determining an analysis rule of the business data according to the data type of the business data to obtain business data features; performing spectral clustering dimension reduction processing and probability clustering on the business data features by using a dynamic cataloging model to determine business field information of the business data, performing context coding on the business data features to determine associated information between the business data, and determining a second classification result of the business data according to the business field information and the associated information; and determining a data cataloging result of the business data according to the first classification result and the second classification result. The application can solve the problem of insufficient modeling of complex inter-field dependency relationships in traditional methods.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a data cataloging method, system, device and storage medium based on artificial intelligence. Background Technology

[0002] Intelligent cataloging is a process that uses artificial intelligence technology to automatically classify, label, and catalog multimedia content such as documents and images. Based on text classification and character recognition technologies, it analyzes the content and structure of documents, automatically categorizes them, and generates corresponding catalog information.

[0003] In the data-driven era, efficient and accurate management and integration of massive and heterogeneous data resources are crucial. Data cataloging, as a core component of data governance, directly impacts data discoverability, usability, and value extraction through its efficiency and accuracy. In recent years, automated cataloging tools based on semi-automated machine learning methods have been widely adopted due to their potential to improve cataloging efficiency. These tools typically employ a human-machine collaboration model, with a typical process as follows: First, basic metadata is automatically obtained by parsing file header information and database schemas; second, the data is initially classified using preset keyword matching rules or format recognition rules (e.g., files containing "financial report" are classified as "financial," and JPEG files are classified as "image resources"); finally, traditional clustering algorithms or shallow neural networks are applied to the structured data for feature extraction. Throughout this process, the construction and maintenance of the classification tree and label system are highly dependent on manual intervention, a typical example being the static classification module of Apache Atlas.

[0004] Traditional automated cataloging tools based on semi-automated machine learning methods mainly suffer from the following problems:

[0005] I. Insufficient Accuracy of Keyword-Based Automated Cataloging: Relying solely on keyword matching and simple format recognition rules makes it difficult to accurately understand the semantics and contextual information of data files. For data files with complex content or multiple semantic meanings, classification errors or inability to classify them are prone to occur.

[0006] Second, the need for manual maintenance creates efficiency bottlenecks: the classification tree and label system require continuous manual maintenance, the rules need to be reconstructed when new data types are added, staff are prone to cataloging errors due to fatigue, and it is difficult to cope with the rapid growth of data.

[0007] Third, the data format types processed are relatively limited: Semi-automated machine learning methods have poor performance in processing image data (such as PDFs and images) and emerging data formats (such as multimodal data and real-time streaming data), requiring frequent manual adjustments to classification rules and resulting in high maintenance costs. Summary of the Invention

[0008] This application aims to at least solve the technical problems existing in the prior art and provide a data cataloging method, system, device and storage medium based on artificial intelligence.

[0009] In a first aspect, the present invention provides a data cataloging method based on artificial intelligence, comprising:

[0010] Obtain the business data to be processed;

[0011] Modality recognition and keyword extraction are performed on business data to obtain the first classification result and data type of the business data;

[0012] Based on the data type of the business data, determine the parsing rules for the business data, and extract features from the business data according to the determined parsing rules to obtain the business data features;

[0013] The business data features are input into the dynamic cataloging model. The dynamic cataloging model performs spectral clustering dimensionality reduction and probabilistic clustering on the business data features to determine the business domain information of the business data. It also performs context encoding on the business data features to determine the correlation information between the business data. Finally, based on the business domain information and the correlation information between the business data, it determines the second classification result of the business data.

[0014] The data cataloging results for business data are determined based on the results of the first and second classifications.

[0015] Secondly, the present invention provides a data cataloging system based on artificial intelligence, the system comprising:

[0016] The acquisition module retrieves the business data to be processed.

[0017] The first processing module is used to perform modality recognition and keyword extraction on business data to obtain the first classification result and data type of the business data;

[0018] The feature extraction module is used to determine the parsing rules of business data based on the data type of business data, and to extract features from the business data according to the preset parsing rules to obtain business data features;

[0019] The second processing module is used to input business data features into the dynamic cataloging model. The dynamic cataloging model performs spectral clustering dimensionality reduction and probabilistic clustering on the business data features to determine the business domain information of the business data, performs context encoding on the business data features to determine the correlation information between business data, and determines the second classification result of the business data based on the business domain information and the correlation information between each business data.

[0020] The output module is used to determine the data cataloging results of business data based on the first classification result and the second classification result.

[0021] Thirdly, the present invention provides an electronic device, the electronic device comprising:

[0022] At least one processor; and,

[0023] A memory communicatively connected to the at least one processor; wherein,

[0024] The memory stores a computer program that can be executed by the at least one processor, which enables the at least one processor to perform the artificial intelligence-based data cataloging method described above.

[0025] Fourthly, the present invention also provides a computer-readable storage medium storing at least one computer program, which is executed by a processor in an electronic device to implement the artificial intelligence-based data cataloging method described above.

[0026] In summary, this application includes the following beneficial technical effects:

[0027] A two-stage hierarchical strategy is adopted for intelligent catalog generation. The first stage achieves coarse-grained partitioning through spectral clustering and Gaussian mixture model, and the second stage uses Transformer and relational network for fine-grained classification, which effectively solves the problem of insufficient modeling of complex dependencies between fields in traditional methods.

[0028] Through a multimodal data adaptive processing mechanism, it achieves unified processing of structured data, unstructured data, and time-series data, breaking through the limitation of traditional cataloging methods on a single data format type;

[0029] Through knowledge graph-driven semantic association, a three-dimensional knowledge association system has been constructed, including a business ontology library, a data lineage graph, and a scenario-based association engine, forming a closed-loop collaborative system of "semantic definition - data context - scenario application". The system can automatically identify business entities scattered across different systems, establish field-level semantic associations, eliminate information silos, and achieve deep collaboration of multi-source heterogeneous data, greatly improving the efficiency of data cataloging. Attached Figure Description

[0030] Figure 1 This is a flowchart illustrating an artificial intelligence-based data cataloging method according to an embodiment of the present invention.

[0031] Figure 2 This is a schematic diagram of the structure of an electronic device that implements the artificial intelligence-based data cataloging method according to an embodiment of the present invention.

[0032] Reference numerals: 10, processor; 11, memory; 12, communication bus; 13, communication interface.

[0033] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0034] Embodiments of the present invention are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.

[0035] In the description of this invention, it should be understood that the terms "longitudinal", "lateral", "up", "down", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer", etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this invention.

[0036] In the description of this invention, unless otherwise specified and limited, it should be noted that the terms "installation", "connection" and "linking" should be interpreted broadly. For example, they can refer to mechanical or electrical connections, or internal connections between two components. They can be direct connections or indirect connections through an intermediate medium. Those skilled in the art can understand the specific meaning of the above terms according to the specific circumstances.

[0037] Reference Figure 1 The diagram shown is a flowchart illustrating an artificial intelligence-based data cataloging method according to an embodiment of the present invention. In this embodiment, the artificial intelligence-based data cataloging method includes:

[0038] S1. Obtain the business data to be processed.

[0039] Specifically, the AI-based data cataloging method in this embodiment first acquires the business data to be processed through a data acquisition interface. This business data can come from various internal business systems of the enterprise, such as customer relationship management systems, enterprise resource planning systems, and supply chain management systems, or from external data sources, such as public datasets and third-party data service providers. The acquired business data may contain various formats and types, such as text documents, images, audio, video, and database tables.

[0040] The data types of business data include at least one of structured data, unstructured data, and time-series data.

[0041] In this embodiment, business data refers to data generated by an enterprise during its daily operations that is related to specific business activities. Business data is used to record detailed information and execution status of various business activities of the enterprise, such as sales orders, purchase orders, inventory inbound and outbound records, production work orders, etc.

[0042] Structured data refers to data that can be represented and stored using relational databases, typically in the form of two-dimensional tables. Structured data includes data in relational databases (such as customer data, order data, product data, etc.) and tabular data (such as financial statements, etc.). Each row of structured data represents information about an entity, and each column of data has strict homogeneity.

[0043] Unstructured data refers to data without a fixed structure. It typically includes documents in various formats related to a company's business (such as human resources documents, operations management records, market analysis reports, project management documents, legal compliance documents, etc.), images (such as scanned copies of contracts, organizational charts), videos (such as product introduction videos), and audio (such as meeting minutes recordings). Unstructured data does not have fixed fields or types.

[0044] Time series data refers to a sequence of data recorded in chronological order, with each data point associated with a specific timestamp, used to describe the state or extent of a phenomenon's change over time; examples include transaction record analysis data, stock price data, and vehicle location data.

[0045] S2. Perform modality recognition and keyword extraction on the business data to obtain the first classification result and data type of the business data.

[0046] After acquiring business data, modality recognition and keyword extraction are performed. Modality recognition refers to identifying the presentation format of business data, such as text, images, audio, and video. In this embodiment, a deep learning model is used for modality recognition. This model has been trained on a large amount of multimodal data and can accurately identify the modality type of business data. Keyword extraction involves extracting keywords or phrases that represent the core content of the business data. For text data, a combination of the TF-IDF (Term Frequency-Inverse Document Frequency) algorithm and the TextRank algorithm is used to extract keywords. For image data, a convolutional neural network is used to extract visual features and identify key content. For audio and video data, speech recognition or video analysis is used to convert the data into text or images before keyword extraction. Through modality recognition and keyword extraction, the first classification result and data type of the business data are obtained. The first classification result is a preliminary classification based on modality and keywords, such as classifying text documents containing the keyword "financial statement" as financial documents.

[0047] S3. Determine the parsing rules for business data based on the data type of the business data, and extract features from the business data according to the determined parsing rules to obtain business data features.

[0048] Specifically, the parsing rules for business data are determined based on the data type, and features are extracted from the business data according to the preset parsing rules to obtain business data features. The data types of business data include at least one of structured data, unstructured data, and time-series data. Structured data refers to data with a predefined schema, such as database tables; unstructured data refers to data without a predefined schema, such as text documents and images; time-series data refers to data sequences that change over time, such as stock prices and sensor readings.

[0049] For structured data, the parsing rule is as follows: Parse the metadata structure of the business data and extract field features. Then, generate numerical representations using a feature vectorization engine to obtain the business data features. Specifically, first, analyze the table structure, field types, field names, and other metadata information. Next, extract information such as relationships between fields, data distribution characteristics, and the business meaning of the fields. Finally, convert these features into numerical vector representations using the feature vectorization engine. The feature vectorization engine employs algorithms such as Word2Vec and FastText to convert text information such as field names and data types into vector representations, and combines these with statistical features (such as mean, variance, and skewness) to form a complete feature representation.

[0050] For unstructured data, the parsing rule is as follows: extract spatial and textual features from the business data, and then fuse these features to obtain the business data features. For text-based unstructured data, pre-trained language models such as BERT (Bidirectional Encoder Representations from Transformers) and GPT (Generative Pretrained Transformer) are used to extract textual features. For image-based unstructured data, convolutional neural networks such as ResNet (Residual Network) and VGG (Visual Geometry Group) are used to extract spatial features. For mixed-type data containing both text and images, textual and spatial features are extracted separately, and then feature fusion is performed using an attention mechanism to obtain a comprehensive feature representation.

[0051] For time-series data, the parsing rule is as follows: A bidirectional Long Short-Term Memory (Bi-LSTM) network is used to extract local and global time-series features of the business data, respectively. Based on these local and global time-series features, the business data features are obtained. Specifically, the time-series data is first preprocessed, including missing value imputation, outlier handling, and standardization. Then, a Bi-LSTM network is used to process the preprocessed time-series data. One LSTM layer extracts local time-series features (such as short-term trends and periodic changes), while the other LSTM layer extracts global time-series features (such as long-term trends and seasonal changes). Finally, the local and global time-series features are fused to obtain a complete representation of the time-series data features. LSTM stands for Long Short-Term Memory.

[0052] S4. Input the business data features into the dynamic cataloging model. The dynamic cataloging model performs spectral clustering dimensionality reduction and probabilistic clustering on the business data features to determine the business domain information of the business data. It also performs context encoding on the business data features to determine the correlation information between the business data. Based on the business domain information and the correlation information between the business data, it determines the second classification result of the business data.

[0053] The dynamic cataloging model is a multi-task learning framework that includes a spectral clustering dimensionality reduction module, a probabilistic clustering module, and a context encoding module. The spectral clustering dimensionality reduction module performs dimensionality reduction processing on business data features to reduce feature dimensions and improve the efficiency of subsequent processing. The probabilistic clustering module performs cluster analysis on the dimensionality-reduced features to determine the business domain information of the business data. The context encoding module performs context encoding on the business data features to determine the correlation information between business data.

[0054] Specifically, the spectral clustering dimensionality reduction module first constructs a similarity matrix of data features and calculates the cosine similarity or Euclidean distance between feature vectors. Then, it constructs a Laplacian matrix based on the similarity matrix and calculates the eigenvalues ​​and eigenvectors of the Laplacian matrix. Finally, it selects the eigenvectors corresponding to the k smallest eigenvalues ​​as the feature representations after dimensionality reduction. The probabilistic clustering module uses a Gaussian Mixture Module (GMM) for clustering. Compared to the traditional K-means algorithm, GMM can provide a probability distribution for clustering, making it more suitable for handling complex business data. GMM iteratively optimizes using the Expectation-Maximization (EM) algorithm to determine the probability of each data point belonging to each cluster, thereby determining the business domain information of the business data.

[0055] The steps for context-encoding business data features to determine the relationships between business data include:

[0056] Business data features are input into the Transformer model, which uses a self-attention mechanism to encode the business data features in context, thereby obtaining contextual relationship information of the business data. Based on the contextual relationship information of the business data, a relationship network is constructed to obtain the association information between business data.

[0057] Specifically, the Transformer model includes a multi-head self-attention layer and a feedforward neural network layer. The multi-head self-attention layer calculates the attention weights between feature vectors, capturing the dependencies between features; the feedforward neural network layer performs a non-linear transformation on the output of the attention layer, enhancing the model's expressive power. Through the processing of the Transformer model, a feature representation considering contextual information is obtained.

[0058] The computation process of the multi-head self-attention mechanism is as follows: First, the input features are linearly mapped into a query vector, a key vector, and a value vector. Then, the dot product of the query vector and the key vector is calculated, and scaling and softmax normalization are applied to obtain the attention weights. Finally, the attention weights are multiplied by the value vectors and summed to obtain the weighted feature representation. The multi-head self-attention mechanism executes the above process in parallel multiple times, using a different linear mapping each time, and then concatenates the multiple results, obtaining the final output through another linear mapping.

[0059] The feedforward neural network sublayer is a two-layer fully connected network. The first layer uses the ReLU activation function, and the second layer is a linear mapping. The feedforward neural network sublayer processes features at each location independently and does not share parameters.

[0060] The Transformer model also includes residual connections and layer normalization. Residual connections alleviate the gradient vanishing problem in deep networks, while layer normalization accelerates the model's convergence speed. Through the processing of multiple Transformer encoders, business data features are mapped to a feature space that considers contextual information, thus obtaining the contextual relationship information of the business data.

[0061] A relationship network is constructed based on the contextual relationships of business data to obtain the association information between business data. The relationship network is a weighted directed graph where nodes represent business data, edges represent the relationships between data, and edge weights represent the strength of the association. The process of constructing the relationship network includes: First, calculating the similarity between contextual features of business data, using metrics such as cosine similarity and dot product similarity; second, setting a threshold based on the similarity; if the similarity between two pieces of business data exceeds the threshold, a connection is established between them, with the connection weight being the similarity value; finally, the relationship network is optimized, such as removing redundant connections and merging similar nodes, to obtain a simplified relationship network.

[0062] Relationship networks can be represented using adjacency matrices or adjacency lists, facilitating subsequent graph analysis operations. Through relationship networks, direct and indirect connections between business data can be discovered, such as data dependencies, data flows, and data groups. This information is invaluable for understanding the business context of the data and optimizing data management strategies.

[0063] In practical applications, relationship networks can be visualized to help users intuitively understand the relationships between data. Relationship networks can also serve as input to graph neural networks to further extract graph structure features for downstream tasks such as anomaly detection and recommendation systems.

[0064] A relationship network is constructed based on the contextual information of business data to obtain the association information between business data. The relationship network is a graph structure where nodes represent business data, edges represent the relationships between data, and edge weights indicate the strength of the association. The construction of the relationship network is based on contextual information; if the similarity of the contextual features of two pieces of business data exceeds a preset threshold, a connection is established between them. Through relationship network analysis, implicit associations between business data can be discovered, such as data dependencies and data flow.

[0065] The dynamic cataloging model determines a second classification result for business data based on business domain information and the correlation information between various business data. This second classification result is a refined classification based on deep features and relationships, and is more accurate and comprehensive than the first classification result. Specifically, the dynamic cataloging model comprehensively considers business domain information (such as which business domain the data belongs to) and correlation information (such as which other data the data is associated with), and determines the final classification result through a weighted fusion method.

[0066] S5. Determine the data cataloging results of the business data based on the first and second classification results.

[0067] Data cataloging results include information such as data category labels, data descriptions, and data relationships. Specifically, if the first and second category results are consistent, the first category result is used directly; otherwise, a weighted fusion is performed based on preset weights, with the second category result typically having a higher weight because it is based on deeper feature and relationship analysis. Data cataloging results can be used in various application scenarios such as data management, data retrieval, and data analysis.

[0068] In a preferred embodiment of this example, the artificial intelligence-based data cataloging method further includes:

[0069] S401. Use an adaptive encoder to dynamically encode and optimize the business data features to obtain optimized business data features.

[0070] Specifically, an adaptive encoder is used to dynamically encode and optimize the business data features to obtain optimized business data features, including:

[0071] S410. Use an adaptive encoder to perform dynamic feature encoding on the business data features to obtain the image features and text features of the business data.

[0072] Specifically, the adaptive encoder first preprocesses the input business data features, including normalization and denoising; then, the encoder maps the preprocessed features to the latent space to obtain the latent representation; finally, two independent decoders generate image features and text features respectively. Image features mainly capture the visual patterns and spatial structure of the data, while text features mainly capture the semantic information and contextual relationships of the data.

[0073] S420. Utilize image and text features to perform multimodal comparative learning and extract multimodal relationship information from business data.

[0074] Multimodal contrastive learning is a self-supervised learning method that learns a unified cross-modal representation by maximizing the mutual information between different modal representations of the same data and minimizing the mutual information between different data representations. Specifically, for each business data point, its image and text features are considered positive sample pairs, while its features with those of other business data are considered negative sample pairs. The model parameters are optimized using a contrastive loss function so that the similarity of positive sample pairs is higher than that of negative sample pairs. Through multimodal contrastive learning, multimodal relationship information can be extracted from business data, namely, the correspondence and complementary information between different modalities.

[0075] S430. Optimize business data features based on multimodal relationship information to obtain optimized business data features.

[0076] Specifically, multimodal relationship information is used as attention weights to perform weighted fusion of the original business data features, highlighting important features and suppressing noisy features. The optimized business data features combine the advantages of image and text features, resulting in stronger expressive and discriminative capabilities.

[0077] Specifically, an adaptive encoder is a neural network model based on a variational autoencoder (VAE) that automatically adjusts encoding parameters according to the characteristics of the input data, adapting to different types of data features. An adaptive encoder consists of two parts: an encoder and a decoder. The encoder maps input features to a latent space, and the decoder reconstructs the original features from the representation in the latent space. Unlike traditional VAEs, adaptive encoders introduce attention mechanisms and residual connections, enhancing the model's expressive power and generalization ability.

[0078] In this embodiment, the adaptive encoder employs a scalable neural network architecture. Its core lies in its ability to dynamically adjust the network structure based on the feature dimensions of the input data. The implementation mechanism is as follows: The encoder consists of multiple independent neuron modules, each containing a fully connected layer, an activation function (such as Swish), and weight parameters. Initially, the network contains only basic modules (e.g., 512 neurons). When the system detects new features or significant changes in feature distribution in the input data (determined by KL divergence > 0.15), it automatically triggers expansion. The weights of the newly added modules are not randomly initialized but inherit the average weights of existing modules, avoiding performance fluctuations caused by initialization differences. By monitoring changes in data feature dimensions online (such as adding business fields), the number of hidden layer neurons and connection weights of the encoder are dynamically adjusted, designing a dynamic weight matrix W∈R. m×n Where m is the input dimension and n is the hidden layer dimension. When the input dimension increases from m to m', W is expanded to W'∈R through zero-padding and historical weight interpolation. m’×n’ This ensures forward compatibility while avoiding dimensional mismatch issues inherent in traditional fixed architectures.

[0079] In multimodal contrastive learning, a triplet (anchor sample, positive sample, negative sample) is first constructed. The feature distance between samples of different modalities with the same semantics is narrowed by the contrastive loss function. This allows the model to continuously absorb new data through incremental learning without forgetting old knowledge, and adapt to the dynamic changes in data distribution. This enables the model to identify model parameters that are important to the historical task when updating parameters shared between new and old tasks, preventing them from being over-modified by the new task and controlling the performance degradation of the historical task.

[0080] Next, a cache is built to store a fixed number of historical samples (e.g., 50 samples per class). When a new sample is added, the old sample with the lowest information content is replaced according to the information content score. When the classification accuracy of historical data is detected to drop below a threshold, historical data playback is automatically triggered. A portion of samples (e.g., 20%) is randomly selected from the buffer and mixed with the new data for retraining, ensuring that the model learns both new and old knowledge at the same time.

[0081] For samples with classification confidence, they are automatically pushed to a manual annotation platform, employing an "uncertainty sampling + diversity sampling" strategy. Uncertainty sampling targets samples whose model predictions are ambiguous and difficult to distinguish, while diversity sampling aims to ensure that the selected samples cover different regions of the data distribution, avoiding duplicate annotation of similar data. First, a candidate set with high information content is screened out through uncertainty, and then samples covering diversity are selected from it.

[0082] Specifically, the process of screening candidate sets under uncertainty includes:

[0083] 1. Model prediction and uncertainty quantification:

[0084] For an unlabeled dataset U = {x1, x2, x3, ..., x...} n Perform batch predictions and generate class probability distributions. Where i represents the index of the data in the unlabeled dataset.

[0085] Calculate the uncertainty index for each sample, which includes entropy and margin.

[0086] 2. Dynamic threshold truncation:

[0087] Set the uncertainty threshold to θ H :

[0088] θ H =μ H +ασ H

[0089] Where μH is the average entropy, σH is the standard deviation, and α∈[1.0,2.5] is the adjustment coefficient. When the uncertainty index of the sample is greater than or equal to the uncertainty threshold θ, H If the selection criteria are met, the sample that meets the criteria will be placed into the uncertainty screening candidate set.

[0090] The steps for selecting diversity samples include:

[0091] 1. Feature space mapping:

[0092]

[0093] Map the uncertainty-filtered candidate set Cuncertain to the feature space:

[0094] Where, x i To filter sample indices within the candidate set for uncertainty, f i The deep feature vectors corresponding to samples in the candidate set are selected for uncertainty, where d represents the feature dimension.

[0095] For {f i Perform K-means clustering:

[0096]

[0097] Among them, g k Describing a cluster, μ k Let K be the cluster center, K be the number of elements in the cluster, k be the index of an element in the cluster, ||.||2 represent the second normal form, and arg min represent the value of the independent variable that makes the function reach its minimum value.

[0098] For each cluster g k Select representative samples, and choose samples close to the cluster center. and boundary samples (covering the edge of the decision space) The final sample set is obtained:

[0099]

[0100] The algorithm described above maximizes model performance improvement with minimal manual annotation cost; it selects the most valuable labeled samples to reduce annotation costs.

[0101] S402. Input the optimized business data features into the dynamic cataloging model to obtain the second classification result.

[0102] Because the optimized features are of higher quality and contain more information, the resulting second classification results are more accurate and reliable.

[0103] In another embodiment of this example, the step of context-encoding business data features to determine the association information between business data includes:

[0104] S6. Construct a data lineage graph based on image features, text features, and multimodal relationship information, and store the business data and the data lineage graph together.

[0105] A data lineage graph is a special type of knowledge graph that records the source, flow, and evolution of data. Specifically, nodes in a data lineage graph represent business data, and edges represent the lineage relationships between data, such as derivation and combination relationships. The process of constructing a data lineage graph includes: first, calculating the similarity between data based on image and text features; then, determining the type and strength of the lineage relationships between data based on similarity and multimodal relationship information; and finally, organizing the lineage relationships into a graph structure to form a complete data lineage graph.

[0106] Data lineage diagrams are stored in association with business data, facilitating subsequent applications such as data tracing and data impact analysis. This association storage is implemented using graph databases, such as Neo4j and JanusGraph, supporting efficient graph query and analysis operations. Through data lineage diagrams, users can intuitively understand the source and flow of data, improving the transparency and traceability of data governance.

[0107] In a preferred embodiment of this example, the method further includes constructing a three-dimensional knowledge association system.

[0108] The three-dimensional knowledge association system consists of a business ontology (acquired business data), a data lineage graph, and a scenario-based association engine. These three components form a closed-loop collaborative system of "semantic definition—data context—scenario application." The business ontology assigns business semantics to data fields, making lineage relationships understandable to business personnel. The data lineage graph provides data credibility assessment (e.g., whether the field source is authoritative), guiding the scenario-based association engine to prioritize highly reliable data in specific scenarios. The scenario-based association engine dynamically updates and optimizes the business ontology and lineage rules based on user feedback (e.g., annotation and error correction).

[0109] A business ontology library facilitates terminology standardization. It uses OWL (Web Ontology Language) to define ontology classes, attributes, and relationships, and implements ontology reasoning through the Jena framework. This maps homonymous or synonymous fields scattered across different systems to a unified business concept, eliminating semantic ambiguity.

[0110] Data lineage graphs reduce the lifecycle of managed data, track upstream and downstream dependencies of data tables, store field-level lineage links through a graph database, and support reverse tracing queries. By parsing SQL, ETL scripts, and API logs, a field-level lineage graph is constructed, recording the entire data flow path from acquisition (e.g., Kafka real-time streaming), processing (e.g., Spark ETL), to application (e.g., BI reports). Data credibility tags are attached to catalog entries. When the structure of a field changes, the downstream catalog entries that depend on that field are automatically located, triggering incremental updates. The graph is updated in real time through a hook mechanism.

[0111] The scenario-based association engine is used to provide scenario-based modeling. In specific business scenarios, based on user roles and historical behaviors, it calculates the similarity of data nodes through graph neural networks. The formula for calculating the similarity of data nodes is:

[0112] Sim(u, v) = σ(W·|h u ||h v |)

[0113] h u h v The node embedding vector is W, which is a trainable parameter. It automatically associates similar data scattered across different systems, dynamically recommends relevant data assets, generates composite features for model training, and provides ontology-driven semantic retrieval services by integrating the SPARQL (Protocol and RDFQuery Language) query engine.

[0114] Knowledge graph-based semantic association technology enables deep collaboration of multi-source heterogeneous data by constructing cross-system data lineage networks and ontology mapping relationships. The system automatically identifies business entities scattered across different systems, establishes field-level semantic relationships, and eliminates information silos.

[0115] Based on the same inventive concept, one embodiment of the present invention provides an artificial intelligence-based data cataloging system.

[0116] The AI-based data cataloging system described in this invention can be installed in an electronic device. According to its functions, the AI-based data cataloging system includes an acquisition module, a first processing module, a feature extraction module, a second processing module, and an output module. The acquisition module acquires business data to be processed; the first processing module performs modality recognition and keyword extraction on the business data to obtain a first classification result and data type; the feature extraction module determines the parsing rules for the business data based on its data type and extracts features from the business data according to the determined parsing rules; the second processing module inputs the business data features into a dynamic cataloging model, which performs spectral clustering for dimensionality reduction and probabilistic clustering to determine the business domain information of the business data, performs context encoding on the business data features to determine the association information between business data, and determines a second classification result of the business data based on the business domain information and the association information between each business data; the output module determines the data cataloging result of the business data based on the first and second classification results.

[0117] The module described in this invention can also be called a unit, which refers to a series of computer program segments that can be executed by the processor of an electronic device and can perform a fixed function, and are stored in the memory of the electronic device.

[0118] The various variations and specific examples of the AI-based data cataloging method provided in the above embodiments are also applicable to the AI-based data cataloging system of this embodiment. Through the foregoing detailed description of the AI-based data cataloging method, those skilled in the art can clearly understand the implementation method of the AI-based data cataloging system in this embodiment. For the sake of brevity, they will not be described in detail here.

[0119] This application also discloses an electronic device, such as Figure 2 The diagram shown is a schematic representation of an electronic device using an artificial intelligence-based data cataloging method according to an embodiment of the present invention. The electronic device may include at least one processor 10, a memory 11 communicatively connected to the at least one processor, a communication bus 12, and a communication interface 13. It may also include a computer program stored in the memory 11 and executable on the processor 10, such as an artificial intelligence-based data cataloging method program.

[0120] In some embodiments, the processor 10 may be composed of integrated circuits, such as a single packaged integrated circuit or multiple integrated circuits with the same or different functions, including combinations of one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and various control chips. The processor 10 is the control unit of the electronic device, connecting various components of the entire electronic device through various interfaces and lines. It executes programs or modules stored in the memory 11 (e.g., executing artificial intelligence-based data cataloging methods) and calls data stored in the memory 11 to perform various functions of the electronic device and process data.

[0121] The memory 11 includes at least one type of readable storage medium, including flash memory, portable hard drive, multimedia card, card-type memory (e.g., SD or DX memory), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 11 can be an internal storage unit of an electronic device, such as a portable hard drive. In other embodiments, the memory 11 can be an external storage device of the electronic device, such as a plug-in portable hard drive, Smart Media Card (SMC), Secure Digital (SD) card, Flash Card, etc. Furthermore, the memory 11 can include both internal and external storage units of the electronic device. The memory 11 can be used not only to store application software and various types of data installed on the electronic device, such as the code of data cataloging methods based on artificial intelligence, but also to temporarily store data that has been output or will be output.

[0122] The communication bus 12 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This bus can be divided into an address bus, a data bus, a control bus, etc. The bus is configured to enable communication between the memory 11 and at least one processor 10, etc.

[0123] Communication interface 13 is used for communication between the aforementioned electronic device and other devices, including a network interface and a user interface. Optionally, the network interface may include a wired interface and / or a wireless interface (such as a Wi-Fi interface, Bluetooth interface, etc.), typically used to establish communication connections between the electronic device and other electronic devices. The user interface may be a display, an input unit (such as a keyboard), or, optionally, a standard wired or wireless interface. Optionally, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, or an OLED (Organic Light-Emitting Diode) touchscreen, etc. The display may also be appropriately referred to as a screen or display unit, used to display information processed in the electronic device and to display a visual user interface.

[0124] Figure 2 Only electronic devices with components are shown; it will be understood by those skilled in the art that... Figure 2The structure shown does not constitute a limitation on the electronic device and may include fewer or more components than shown, or combine certain components, or have different component arrangements. For example, although not shown, the electronic device may also include a power supply (such as a battery) to power the various components. Preferably, the power supply can be logically connected to at least one processor 10 through a power management device, thereby enabling functions such as charging management, discharging management, and power consumption management. The power supply may also include one or more DC or AC power supplies, recharging devices, power fault detection circuits, power converters or inverters, power status indicators, and other arbitrary components. The electronic device may also include various sensors, Bluetooth modules, Wi-Fi modules, etc., which will not be described in detail here.

[0125] It should be understood that the embodiments are for illustrative purposes only and are not limited to this structure in the scope of the patent application.

[0126] Furthermore, if the modules / units integrated into the electronic device are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. The computer-readable storage medium can be volatile or non-volatile.

[0127] This application provides a computer-readable storage medium, including, for example, any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, or a read-only memory (ROM). The computer-readable storage medium stores a computer program capable of being loaded by a processor and executing the artificial intelligence-based data cataloging method of the above embodiments.

[0128] In the description of this specification, the references to terms such as "an embodiment," "some embodiments," "example," "specific example," "a implementation," "a preferred implementation," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0129] Although embodiments of the invention have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the claims and their equivalents.

Claims

1. A data cataloging method based on artificial intelligence, characterized in that, The method includes: Obtain the business data to be processed; Modality recognition and keyword extraction are performed on business data to obtain the first classification result and data type of the business data; Based on the data type of the business data, determine the parsing rules for the business data, and extract features from the business data according to the determined parsing rules to obtain the business data features; The dynamic cataloging model inputs business data features into itself. It performs spectral clustering for dimensionality reduction and probabilistic clustering to determine the business domain information of the business data. It also performs context encoding on the business data features to determine the relationships between them. Finally, based on the business domain information and the relationships between the business data, it determines the second classification result of the business data. The dynamic cataloging model is a multi-task learning framework that includes a spectral clustering dimensionality reduction module, a probabilistic clustering module, and a context encoding module. The spectral clustering dimensionality reduction module performs dimensionality reduction on the business data features, the probabilistic clustering module performs cluster analysis on the dimensionality-reduced features to determine the business domain information of the business data, and the context encoding module performs context encoding on the business data features to determine the relationships between them. The data cataloging results of the business data are determined based on the results of the first and second classifications. The data types of business data include at least one of structured data, unstructured data, and time-series data. The step of determining the parsing rules for business data based on its data type, and extracting features from the business data according to the preset parsing rules to obtain business data features includes: Determine the data type of the business data: If the data type of the business data is structured data, the parsing rules for the business data are as follows: parse the metadata structure of the business data and extract field features, generate numerical representations through the feature vectorization engine, and obtain the business data features; If the data type of the business data is unstructured data, then the parsing rules for the business data are: extract the spatial features and text features of the business data, and fuse the spatial features and text features to obtain the business data features; If the data type of the business data is time series data, then the parsing rule for the business data is: use a bidirectional long short-term memory network to extract the local time series features and global time series features of the business data respectively, and obtain the business data features based on the local time series features and global time series features; The steps for context-encoding business data features to determine the relationships between business data include: The business data features are input into the Transformer model, and the Transformer model uses a self-attention mechanism to perform contextual encoding on the business data features to obtain the contextual relationship information of the business data. A relationship network is constructed based on the contextual relationship information of business data to obtain the association information between business data.

2. The data cataloging method based on artificial intelligence as described in claim 1, characterized in that, The method further includes: An adaptive encoder is used to dynamically encode and optimize the features of business data to obtain optimized business data features. The optimized business data features are input into the dynamic cataloging model to obtain the second classification result.

3. The data cataloging method based on artificial intelligence as described in claim 2, characterized in that, The process of using an adaptive encoder to dynamically encode and optimize business data features to obtain optimized business data features includes: An adaptive encoder is used to dynamically encode the features of business data to obtain image and text features of the business data. Multimodal contrastive learning is performed using image and text features to extract multimodal relationship information from business data; The optimized business data features are obtained by optimizing the business data features based on multimodal relationship information.

4. The data cataloging method based on artificial intelligence as described in claim 1, characterized in that, A data lineage graph is constructed based on image features, text features, and multimodal relationship information, and business data is stored in association with the data lineage graph.

5. An artificial intelligence-based data cataloging system, used to implement the artificial intelligence-based data cataloging method according to any one of claims 1 to 4, characterized in that, include: The acquisition module retrieves the business data to be processed. The first processing module is used to perform modality recognition and keyword extraction on business data to obtain the first classification result and data type of the business data; The feature extraction module is used to determine the parsing rules of business data based on the data type of business data, and to extract features from the business data according to the preset parsing rules to obtain business data features; The second processing module is used to input business data features into the dynamic cataloging model. The dynamic cataloging model performs spectral clustering dimensionality reduction and probabilistic clustering on the business data features to determine the business domain information of the business data, performs context encoding on the business data features to determine the correlation information between business data, and determines the second classification result of the business data based on the business domain information and the correlation information between each business data. The output module is used to determine the data cataloging results of business data based on the first classification result and the second classification result.

6. An electronic device, characterized in that, The electronic device includes: At least one processor (10); and, A memory (11) communicatively connected to the at least one processor (10); The memory (11) stores a computer program that can be executed by the at least one processor (10) to enable the at least one processor (10) to perform the artificial intelligence-based data cataloging method as described in any one of claims 1 to 4.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program; when the computer program is executed by a processor, it implements the artificial intelligence-based data cataloging method as described in any one of claims 1 to 4.

Images