Data processing method and system based on model learning

Abstract

The invention discloses a data processing method and system based on model learning, and relates to the technical field of data processing. The method comprises the following steps: accessing multi-source heterogeneous original data through a distributed framework, and performing format recognition and preprocessing by using a deep learning model to obtain standardized data; inputting the data into the generative AI core layer, and executing metadata management, quality detection and repair, standard fusion and safety protection in parallel by using a pre-trained and fine-tuned model; the high-quality data are stored in the lake and warehouse integrated architecture, and data services are provided; collecting event data in real time, and performing closed-loop optimization and model increment training based on a rule engine and a reinforcement learning algorithm. According to the invention, through fusing the generative AI and reinforcement learning technologies, full-process intelligence and adaptive evolution of data management are realized, and the problems of low automation degree, difficulty in hidden anomaly identification and lack of a closed-loop optimization mechanism in the prior art are effectively solved.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a data processing method and system based on model learning. Background Technology

[0002] As enterprises deepen their digital transformation, data has become a core asset driving business innovation and decision optimization. Faced with exponentially growing, multi-source, heterogeneous data, efficient and accurate governance has become a major challenge for enterprises. Traditional data governance processes typically cover data collection, cleaning, storage, quality testing, and security control, aiming to ensure data consistency, accuracy, and security.

[0003] In existing technologies, data governance primarily relies on manually defined rules and traditional Extract-Transform-Load (ETL) tools. During the data acquisition and preprocessing stages, due to the wide variety of data sources and complex formats, engineers often need to manually write parsing scripts for different data sources. This is not only time-consuming and labor-intensive but also has limited processing capabilities when dealing with unstructured data. Regarding metadata management and quality control, existing solutions heavily depend on manual annotation and rule engines based on fixed thresholds. Operations personnel need to manually trace data lineage and define quality verification rules (such as null checks and range constraints). However, this approach struggles to handle complex business logic conflicts or implicit temporal anomalies, leading to delayed anomaly detection. Furthermore, the remediation process typically requires manual intervention and analysis, resulting in long response times.

[0004] Furthermore, the development of data standards often results in data silos across different business departments, lacking a unified semantic alignment mechanism and hindering cross-departmental data integration. In the field of data security, existing sensitive data identification technologies mostly rely on regular expressions or keyword matching, which can only identify explicit features such as ID numbers and phone numbers. Their recognition rate for semantically implicit sensitive information (such as text containing salary descriptions) is low, and once desensitization strategies are set, they are difficult to dynamically adjust according to usage scenarios, easily leading to over-desensitization or insufficient protection of data. Finally, traditional data governance architectures are mostly unidirectional linear processes, lacking automatic optimization mechanisms based on business feedback. When business dynamics change, governance strategies need to be manually readjusted, resulting in high maintenance costs and difficulty in ensuring the system's continuous adaptability.

[0005] In summary, existing data governance methods, due to their low level of automation, weak ability to identify hidden problems, and lack of closed-loop evolution mechanisms, are no longer sufficient to meet the needs of large-scale, highly complex data management. Therefore, there is an urgent need for a data processing solution that can integrate deep learning and generative technologies to achieve intelligent and adaptive processing throughout the entire process. Summary of the Invention

[0006] This invention provides a data processing method and system based on model learning, solving the problems of low automation, difficulty in identifying latent anomalies, and lack of closed-loop evolutionary mechanisms in existing technologies. To achieve the above objectives, the embodiments of this invention adopt the following technical solutions: In a first aspect, embodiments of the present invention propose a data processing method based on model learning, the method comprising: The system accesses multi-source heterogeneous raw data through a distributed data acquisition framework, and uses a pre-built deep learning classification model and data cleaning algorithm library to perform format recognition and preprocessing on the raw data to obtain standardized preprocessed data. The preprocessed data is input into the generative AI core layer, and multiple pre-trained and fine-tuned generative artificial intelligence models are used to perform intelligent metadata management, intelligent data quality detection and repair, automated data standard fusion, and data security and privacy protection processing in parallel to obtain high-quality data after governance. The high-quality data after treatment is stored in the lake warehouse integrated storage architecture, a full-text search index and data version snapshot are built, and the high-quality data is transformed into data services and provided to users through the application interface layer; Real-time collection of service call data from the application interface layer and event data during the governance process; closed-loop optimization analysis based on rule engine and reinforcement learning algorithm; generation of optimization strategy; incremental training of the model in the generative AI core layer; and obtaining an updated data processing model.

[0007] Preferably, a pre-built deep learning classification model and data cleaning algorithm library are used to perform format recognition and preprocessing on the original data to obtain standardized preprocessed data, including: A deep learning classification model combining convolutional neural network (CNN) and long short-term memory network (LSTM) is used to extract and fuse the file header features and content sequence features of the original data to identify the data format of the original data. Based on the identified data format, the corresponding parsing strategy is invoked to convert the raw data into a unified data stream; The SimHash algorithm is used to calculate the fingerprint features of the data stream and perform deduplication. At the same time, the numerical data is preliminarily screened and filtered for outliers based on the 3σ principle to obtain standardized preprocessed data. Preprocessed data is pushed to the generative AI core layer via a message queue.

[0008] Preferably, multiple pre-trained and fine-tuned generative artificial intelligence models are used to perform intelligent metadata management in parallel, including: The ANTLR4 parser is used to perform syntax tree analysis on the scripts in the preprocessed data to extract physical metadata of table structure, field names and data types. The pre-trained natural language processing model ERNIE 3.0 is invoked, and combined with a pre-built enterprise business dictionary, semantic mapping analysis is performed on the extracted physical metadata to generate standardized metadata tags containing business meaning and data source. Collect read and write operations and interface call relationships from data processing logs, and use graph neural networks (GNNs) to construct a directed graph of data nodes and processing nodes to generate a data lineage graph. When a change in the data link is detected, the data lineage map is refreshed within a preset time window using an incremental update algorithm.

[0009] Preferably, performing intelligent data quality detection and repair includes: A dynamic baseline model is built based on Generative Adversarial Network (GAN). The generator of DCGAN architecture generates simulated normal data based on historical normal data, and the discriminator of CNN architecture distinguishes between real normal data and simulated normal data until the loss function converges, thus determining the normal data baseline. Principal Component Analysis (PCA) algorithm is used to reduce the dimensionality of real-time incoming data and extract key features, and the cosine similarity between the key features and the normal data benchmark is calculated. When the cosine similarity is lower than the preset threshold, the data is judged to be abnormal and an early warning is triggered. The domain knowledge graph is called to perform contextual association analysis on the abnormal data to locate the root cause of the abnormality. Based on entity relationships in the domain knowledge graph, repair rules are generated to automatically complete or correct simple anomalies; for complex anomalies, a visual report containing repair suggestions is generated.

[0010] Preferably, the automated fusion of data standards includes: Using the Few-Shot learning framework, enterprise industry attributes, business scope, and historical data samples are input into a pre-trained BERT model for semantic encoding, generating a draft data standard that includes data format, encoding rules, and value range. The Sentence-BERT model is used to convert existing data standards from different departments into semantic vectors, calculate the semantic similarity of cross-departmental standards, identify standard differences, and generate a difference comparison matrix. Based on the genetic algorithm, a fusion scheme is searched for the identified standard differences. The objective function is to maximize data consistency and minimize business impact. The optimal data standard fusion strategy is then output through iterative optimization. Based on the optimal data standard fusion strategy, the preprocessed data is standardized, mapped, and transformed.

[0011] Preferably, data security and privacy protection processing is performed, including: The RoBERTa-BiLSTM-CRF deep learning model is used to perform semantic-level analysis on the data, identify encrypted data and semantically implicit sensitive information in the data, determine the identified information as sensitive data, and determine the sensitivity level of the sensitive data. Construct a de-identification strategy decision tree based on the data usage scenario, user roles, and permission levels; Based on the desensitization strategy decision tree, the system automatically selects partial masking, field replacement, data generalization, or differential privacy algorithms to dynamically desensitize sensitive data. The attribute-based access control ABAC model binds anonymized data to user attributes to achieve dynamic authorization and access auditing.

[0012] Preferably, the application interface layer transforms high-quality data into data services and provides them to users, including: Deploy a natural language processing gateway and integrate it into the ChatGLM model to receive users' natural language query requests; The natural language query request is converted into a standardized structured query language (SQL) or application programming interface (API) call command, and then sent to the lake warehouse integrated storage architecture for data retrieval; Obtain search results and generate reports or real-time dashboards using a pre-built library of visualization components to provide feedback to users; The service monitoring module collects API call frequency, response time, and error rate metrics in real time, encapsulates them into service performance data, and pushes them into a closed-loop optimization and analysis process.

[0013] Preferably, closed-loop optimization analysis is performed based on a rule engine and reinforcement learning algorithm to generate optimization strategies and incrementally train the model in the core layer of generative AI, including: Use a rule engine to automate the handling of common abnormal events; When encountering unforeseen emergencies, the intelligent contingency plan generation module is activated, and the case reasoning CBR algorithm is used to retrieve similar cases from the historical event database. By combining a standard contingency plan library, reinforcement learning (RL) algorithms are used to iteratively optimize the retrieved cases, generate the optimal response plan, and allocate tasks through service mesh technology. After the incident is handled, a debriefing report is generated and feature data is extracted. Incremental training algorithms are then used to feed the feature data back into the models for metadata management, quality inspection, standards management, and security protection, updating the model parameters and rule base.

[0014] Preferably, the optimization process of the reinforcement learning (RL) algorithm includes: Construct a reinforcement learning environment, defining the state space as the current event characteristics and system resource state, and the action space as the set of disposal measures; Define a reward function, which is constructed based on the processing time, resource consumption, and business recovery level. The reinforcement learning agent is used to learn through trial and error in a reinforcement learning environment. The policy network is updated by maximizing the cumulative reward until the policy converges, and the optimal treatment sequence is output as the optimal treatment plan.

[0015] Secondly, embodiments of the present invention propose a data processing system based on model learning, the system comprising: The data acquisition and preprocessing module is used to access multi-source heterogeneous raw data through a distributed data acquisition framework, and to use a pre-built deep learning classification model and data cleaning algorithm library to perform format recognition and preprocessing on the raw data to obtain standardized preprocessed data. The generative AI core processing module is used to input preprocessed data into the generative AI core layer. It utilizes multiple pre-trained and fine-tuned generative artificial intelligence models to perform intelligent metadata management, intelligent data quality detection and repair, automated data standard fusion, and data security and privacy protection processing in parallel to obtain high-quality data after governance. The storage and service management module is used to store the high-quality data after governance into the lake warehouse integrated storage architecture, build full-text search indexes and data version snapshots, and transform the high-quality data into data services for users through the application interface layer; The closed-loop optimization module is used to collect service call data and event data in the application interface layer in real time and to perform closed-loop optimization analysis based on the rule engine and reinforcement learning algorithm. It generates optimization strategies and incrementally trains the model in the generative AI core layer to obtain the updated data processing model.

[0016] Beneficial effects: This invention provides a model-based data processing method that integrates a distributed data acquisition framework with a pre-built deep learning classification model. This achieves automated format identification and cleaning of multi-source heterogeneous raw data, effectively overcoming the inefficiency of relying on manually written scripts for data access in existing technologies and significantly shortening the data preparation cycle. By constructing a generative artificial intelligence core layer, a pre-trained and fine-tuned large model is used to perform metadata management, quality detection, standard fusion, and security protection tasks in parallel. This mechanism leverages the deep semantic understanding capabilities of generative models, overcoming the limitations of traditional rule engines. It can accurately identify complex implicit data anomalies and semantically sensitive information, and achieve intelligent metadata annotation and data repair, significantly improving the accuracy and automation level of data governance. Furthermore, a closed-loop optimization analysis mechanism based on rule engines and reinforcement learning algorithms is introduced, enabling real-time collection of service feedback and governance event data from the application interface layer for continuous incremental training of the core layer model. This design endows the system with adaptive evolution capabilities, enabling it to dynamically adjust and optimize strategies based on changes in the business environment. This avoids the performance degradation that traditional static governance architectures experience over time, ensuring the efficiency and stability of the data processing model in long-term operation and significantly reducing operational costs and decision-making risks. Attached Figure Description

[0017] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings: Figure 1 This is a flowchart of a model-based data processing method proposed in an embodiment of the present invention; Figure 2 This is a schematic diagram of a model-based data processing system proposed in an embodiment of the present invention; Figure 3 This is a schematic diagram of the structure of a computer device provided in an embodiment of the present invention. Detailed Implementation

[0018] To more clearly illustrate the technical solutions in the embodiments of the invention or the prior art, the invention will be briefly introduced below in conjunction with the accompanying drawings and descriptions of the embodiments or the prior art. Obviously, the following description of the structure of the drawings is only some embodiments of the invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. It should be noted that the description of these embodiments is for the purpose of helping to understand the invention, but does not constitute a limitation on the invention.

[0019] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in sequences other than those illustrated or described herein.

[0020] The first aspect, reference Figure 1 This invention provides a model-based data processing method that can be executed on a computer device. The computer device can be a server, a personal computer (PC), or an embedded model-based data processing system with sufficient computing power. This device typically includes, but is not limited to, a processor, memory, communication interfaces, and input / output devices. The memory stores computer program instructions, and the processor executes these instructions to implement the model-based data processing method of this invention. This method aims to address the problems of low automation, weak identification of hidden problems, and lack of closed-loop evolution mechanisms in existing data governance technologies. By constructing a five-layer closed-loop intelligent architecture and integrating advanced technologies such as Generative Artificial Intelligence (GenAI), deep learning, and reinforcement learning, it achieves fully intelligent and adaptive data processing. Figure 1 As shown, the data processing method based on model learning mainly includes steps S101 to S104.

[0021] S101. Access multi-source heterogeneous raw data through a distributed data acquisition framework, and use a pre-built deep learning classification model and data cleaning algorithm library to perform format recognition and preprocessing on the raw data to obtain standardized preprocessed data.

[0022] In enterprise environments undergoing digital transformation, data sources are extremely diverse, including business databases (such as MySQL and Oracle), log files, application programming interface (API) data streams, and IoT sensor data. These data come in various formats and contain a significant amount of noise and redundant information. To ensure efficient subsequent processing, accurate data collection and preprocessing are essential.

[0023] Specifically, step S101 includes the following sub-steps: S1011. Utilize a distributed data acquisition framework to access multi-source heterogeneous raw data.

[0024] This embodiment employs a distributed data acquisition framework based on Flink CDC (Change Data Capture). This framework can monitor and capture incremental change data from various business databases in real time. It also combines this with tools such as Flume or Logstash to collect log files and utilizes Kafka as a message buffer queue to achieve high-throughput, low-latency access to data across the entire system. For API interface data, it is received via polling or webhooking through a configured unified gateway. The raw data entering the system is not yet uniformly parsed and is in a raw binary or text stream state.

[0025] S1012. A deep learning classification model combining Convolutional Neural Network (CNN) and Long Short-Term Memory (LSTM) is used to extract and fuse the file header features and content sequence features of the original data to identify the data format of the original data.

[0026] Traditional format recognition mainly relies on file extensions or magic numbers, which are prone to failure when dealing with files with missing or tampered extensions. This embodiment pre-configures a hybrid deep learning model to solve this problem.

[0027] First, for each incoming data object (file or data packet), extract its first few bytes (e.g., the first 1024 bytes) as the input sequence. These bytes contain file header information and some content characteristics.

[0028] Next, a CNN model is used to extract local features. CNNs use convolutional layers to perform a sliding window scan on the input sequence, extracting local patterns between bytes (such as specific binary identifiers and encoding format markers). The convolution operation, through multiplication and addition operations between the convolution kernel and local data followed by an activation function, can effectively capture the inherent, localized structured features in the file format.

[0029] Then, LSTM networks are used to extract long-range dependency features. Since file format features are often not only reflected in local byte combinations, but also in the sequential relationships and order of byte sequences (i.e., time sequence features), LSTM, as a special type of Recurrent Neural Network (RNN), can remember historical information in long sequences through input gates, forget gates, and output gates, thereby capturing the implicit format logic in the data stream.

[0030] Finally, the local feature vectors extracted by the CNN and the temporal feature vectors extracted by the LSTM are fused (e.g., through concatenation or weighted summation), and then input into a fully connected layer for classification using the normalized exponential function (Softmax). The output is the probability value of the data object belonging to common data formats such as Excel, JSON, CSV, Parquet, and Avro. The category with the highest probability value is selected as the final recognition result. This method boasts high recognition accuracy and eliminates the need for manual configuration of complex matching rules.

[0031] S1013. Based on the identified data format, call the corresponding parsing strategy to convert the raw data into a unified data stream.

[0032] The system has a pre-built library of parsing strategies for different data formats. Once the data format is determined in step S1012, the system automatically routes and calls the appropriate parser. For example, for data identified as JSON, the JSON parser is called to flatten it into a key-value pair stream; for data identified as Excel or CSV, the table parser is called to convert it into a row-column structured record stream. This step uniformly converts heterogeneous raw data from multiple sources into a standardized data stream format defined internally by the system (e.g., an intermediate format based on Protobuf), eliminating format differences.

[0033] S1014. The SimHash algorithm is used to calculate the fingerprint features of the data stream and perform deduplication. At the same time, the numerical data is preliminarily screened and filtered for outliers based on the 3σ principle to obtain standardized preprocessed data.

[0034] In the data cleaning stage, the focus is on solving the problems of duplicate data and obvious outliers.

[0035] Regarding deduplication, this embodiment uses the SimHash algorithm. SimHash is a locality-sensitive hashing algorithm that can map long text or long data records to fixed-length fingerprints (e.g., 64-bit integers) while maintaining a small Hamming distance between fingerprints of similar data. The specific process is as follows: First, the data records are segmented into words, and each feature word is assigned a weight (e.g., based on word frequency). Second, a hash value is calculated for each feature word. Third, for each bit of the hash value, if the bit is 1, the weight of the feature word is added; if it is 0, the weight is subtracted. Then, the weighted vectors of all feature words are summed to obtain the final vector. Finally, each bit of the final vector is evaluated: if the value of the bit is greater than 0, it is set to 1; if it is less than or equal to 0, it is set to 0, thus obtaining the SimHash fingerprint. By calculating the Hamming distance between the SimHash fingerprints of different data records, if the distance is less than a preset numerical threshold, it is determined to be duplicate or highly similar data and is discarded, keeping only one copy.

[0036] Regarding outlier screening, for numerical fields (such as amount, temperature, count, etc.), this embodiment adopts the 3σ principle (Raida criterion). The system calculates the arithmetic mean and standard deviation of the data stream in real time. For each data point, the absolute value of the difference between its value and the mean is calculated. If this absolute value is greater than three times the standard deviation, the data point is considered to deviate too much from the mean and belongs to an outlier with a very low probability. The system marks and filters it, or replaces it with the mean or median to prevent it from interfering with subsequent model training and analysis.

[0037] S1015. Push the preprocessed data to the generative AI core layer via a message queue.

[0038] The data processed through the above steps becomes standardized preprocessed data. The system distributes this data to the downstream generative AI core layer through high-throughput message queues such as Kafka, ensuring real-time data flow and decoupling.

[0039] S102. Input the preprocessed data into the generative AI core layer, and use multiple pre-trained and fine-tuned generative artificial intelligence models to perform intelligent metadata management, intelligent data quality detection and repair, automated data standard fusion, and data security and privacy protection processing in parallel to obtain high-quality data after governance.

[0040] This step is the core of the invention, replacing traditional manual annotation and rule configuration by introducing GenAI technology. This layer contains four parallel intelligent processing modules.

[0041] Specifically, step S102 includes the following sub-steps: S1021. Utilize multiple pre-trained and fine-tuned generative artificial intelligence models to perform intelligent metadata management in parallel.

[0042] Metadata, or "data about data," is the cornerstone of data governance. Traditional methods rely on manual processing, which is extremely inefficient. This embodiment automates this process through the following methods: S10211. Use the ANTLR4 (Another Tool for Language Recognition 4) parser to perform syntax tree analysis on the scripts in the preprocessed data to extract physical metadata of table structure, field names, and data types.

[0043] For the SQL table creation statements, stored procedure scripts, or ETL (Extract-Transform-Load) scripts that are received, the system uses the ANTLR4 tool to perform lexical and syntactic analysis and construct an Abstract Syntax Tree (AST). By traversing the AST, the system can automatically and accurately extract physical metadata such as table names, column names, field types, and constraints, without requiring manual code reading.

[0044] S10212. Call the pre-trained natural language processing model ERNIE 3.0, and combine it with the pre-built enterprise business dictionary to perform semantic mapping analysis on the extracted physical metadata, and generate standardized metadata tags containing business meaning and data source.

[0045] After extracting the physical metadata, the system needs to understand the business meaning of these fields (i.e., the mapping from technical metadata to business metadata). This embodiment utilizes Baidu's knowledge-enhanced model ERNIE 3.0. This model has been pre-trained on massive amounts of text and knowledge graphs, possessing powerful semantic understanding capabilities.

[0046] The system takes extracted field names (such as "cust_id" and "txn_amt") as input and combines them with the enterprise's pre-built business dictionary (containing industry terminology and abbreviation standards) to construct prompts. The ERNIE 3.0 model analyzes the semantic features and contextual relationships of the field names through an attention mechanism, mapping them to standard business terms (such as "customer number" and "transaction amount"), and generating descriptive metadata tags. For example, for the field "dob", the model can recognize it as an abbreviation of "Date of Birth" and tag it with "date of birth". The accuracy of this process is much higher than that of traditional regular expression matching.

[0047] S10213. Collect the read and write operations and interface call relationships in the data processing log, and use a graph neural network (GNN) to construct a directed graph of data nodes and processing nodes to generate a data lineage graph.

[0048] To clarify the origin and flow of data, the system collects operation logs from various components (database, ETL tool, API gateway) in real time. Each data table and field is viewed as a "data node" in the diagram, and each transformation operation (such as JOIN and AGGREGATE) is viewed as a "processing node".

[0049] These nodes and their connections are modeled using a Generative Neural Network (GNN) model. Through message passing, the GNN aggregates features from neighboring nodes, effectively inferring data flow paths and dependencies. Based on this, the system constructs a global Directed Acyclic Graph (DAG), i.e., a data lineage graph. This graph clearly shows every transformation process of data from source to destination.

[0050] S10214. When a data link change is detected, the data lineage map is refreshed within a preset time window using an incremental update algorithm.

[0051] The data environment is dynamic. When the logs detect the creation of a new table or the modification of an old ETL task, the system does not perform a full graph reconstruction, but instead triggers an incremental update algorithm. This algorithm only recalculates the lineage relationships of the affected nodes and their downstream nodes, and completes the local graph refresh within a preset time window (e.g., 10 minutes) to ensure the real-time nature of lineage information.

[0052] S1022. Utilize multiple pre-trained and fine-tuned generative artificial intelligence models to perform intelligent data quality detection and repair in parallel.

[0053] S10221. A dynamic baseline model is constructed based on Generative Adversarial Networks (GANs). The generator of the Deep Convolutional GAN ​​(DCGAN) architecture generates simulated normal data based on historical normal data. The discriminator of the CNN architecture distinguishes between real normal data and simulated normal data until the loss function converges, thus determining the normal data baseline.

[0054] Traditional quality inspection relies on fixed rules, making it difficult to detect distributed, latent anomalies. This embodiment utilizes a Generative Adversarial Network (GAN) to learn the distribution characteristics of normal data. A GAN consists of two adversarial networks: a generator and a discriminator. The generator uses a DCGAN architecture, receiving a random noise vector and attempting to generate simulated data consistent with the distribution of real normal data. The discriminator uses a CNN architecture, receiving real and simulated data, attempting to distinguish them, and outputting a probability value representing the probability that the input data is real. The training process for both is a minimax game. The training objective is to make the generator deceive the discriminator as much as possible, i.e., minimize the probability that the discriminator correctly identifies simulated data; simultaneously, the discriminator should correctly distinguish between real and simulated data as much as possible, i.e., maximize the accuracy of the distinction. As training progresses, the simulated data generated by the generator becomes increasingly realistic, making it increasingly difficult for the discriminator to distinguish, eventually reaching a Nash equilibrium and the loss function converging. At this point, the discriminator has deeply learned the feature boundaries of normal data and can serve as a baseline model for normal data.

[0055] S10222. Use Principal Component Analysis (PCA) algorithm to perform dimensionality reduction on the real-time incoming data and extract key features, and calculate the cosine similarity between the key features and the normal data benchmark.

[0056] Real-time data is often highly dimensional, making direct comparison computationally intensive. The system first uses the PCA algorithm to reduce the dimensionality of the data. PCA transforms a set of potentially correlated variables into a set of linearly uncorrelated variables (principal components) through orthogonal transformation, selecting the top K principal components with the largest contribution rates as key feature vectors. Cosine similarity is used to calculate similarity. Specifically, the dot product (i.e., the sum of corresponding element-wise products) of the key feature vectors of the real-time data and the baseline feature vectors of the normal data extracted by the trained discriminator is calculated, and then divided by the product of the Euclidean norms (i.e., magnitudes) of the two vectors. This result reflects the directional similarity between the two vectors; the closer the value is to 1, the more similar they are.

[0057] S10223. When the cosine similarity is lower than the preset threshold, the data is determined to be abnormal and an early warning is triggered. The domain knowledge graph is called to perform contextual association analysis on the abnormal data to locate the root cause of the abnormality.

[0058] The preset threshold is set based on business tolerance. If the calculated cosine similarity is lower than this value, it indicates that the real-time data deviates from the normal distribution and is judged as an anomaly. The system then calls the domain knowledge graph and analyzes the context of the anomaly data through entity associations in the graph (such as "order amount" associated with "customer level" and "historical consumption habits"). For example, although a large order may have an abnormal value, it may be reasonable if the associated customer level is a high-level member and the historical consumption is high; otherwise, it may be a case of order fraud or data entry error.

[0059] S10224. Generate repair rules based on entity relationships in the domain knowledge graph to automatically complete or correct simple anomalies; for complex anomalies, generate a visual report containing repair suggestions.

[0060] Based on the graph analysis results, the system automatically generates repair strategies. For simple errors (such as mismatched postal codes and addresses), the system directly corrects them using the standard "address-postal code" entity relationship in the graph; for missing values, the system uses the attributes of related entities to infer and complete the missing values. For complex anomalies such as logical conflicts, the system generates a detailed report showing the chain of causes of the anomaly and suggested repair values ​​for manual confirmation.

[0061] S1023. Utilize multiple pre-trained and fine-tuned generative artificial intelligence models to perform parallel automated fusion of data standards.

[0062] S10231. Using the Few-Shot learning framework, input the enterprise's industry attributes, business scope, and historical data samples into the pre-trained BERT (Bidirectional Encoder Representations from Transformers) model for semantic encoding, and generate a draft data standard that includes data format, encoding rules, and value range.

[0063] The BERT model, through a bidirectional Transformer encoder, is able to deeply understand contextual semantics. Under the Few-Shot framework, the system only needs to input a small number of industry samples, and the BERT model can quickly transfer learn to understand the naming conventions and data characteristics of the industry, and automatically generate an initial draft of data standards (for example, stipulating that "mobile phone number" must be 11 digits).

[0064] S10232. Using the Sentence-BERT model, existing data standards from different departments are converted into semantic vectors, the semantic similarity of cross-departmental standards is calculated, standard differences are identified, and a difference comparison matrix is ​​generated.

[0065] Different departments may have existing, inconsistent standards. Sentence-BERT is a Siamese network variant of BERT, specifically designed to generate semantic vectors for sentences. It converts the standard definition texts of different departments into high-dimensional vectors and calculates the cosine similarity between the vectors. If the similarity is high but the wording is different, it is identified as different names (differences) of the same business object, and the system generates a matrix to record these differences.

[0066] S10233. Based on the Genetic Algorithm (GA), a fusion scheme is searched for the identified standard differences. The objective function is to maximize data consistency and minimize business impact. The optimal data standard fusion strategy is output through iterative optimization.

[0067] To find a unified standard acceptable to all departments, this embodiment employs a genetic algorithm. The process includes: first, encoding the solution for each point of discrepancy into a gene sequence (chromosome); then, randomly generating multiple fusion schemes as the initial population. In the evaluation phase, a fitness function (i.e., the objective function) is defined to evaluate the merits of the schemes. This fitness function is a weighted combination of data consistency score and business impact score. Specifically, fitness is directly proportional to data consistency (i.e., a low proportion of data conflicts after unification) and inversely proportional to business impact (i.e., the number of system interfaces that need to be modified and the amount of historical data migration) (or directly proportional to the complement of business impact). Based on the fitness, a superior scheme is selected, and gene crossover and mutation are performed to generate the next generation of schemes. The above process is repeated until the fitness converges, outputting the optimal fusion strategy.

[0068] S10234. Based on the optimal data standard fusion strategy, perform standardized mapping and transformation on the preprocessed data.

[0069] The system automatically generates ETL mapping rules based on the optimal strategy, converting the flowing data into a unified standard in real time.

[0070] S1024. Utilize multiple pre-trained and fine-tuned generative artificial intelligence models to perform data security and privacy protection processing in parallel.

[0071] S10241. Use the RoBERTa-BiLSTM-CRF deep learning model to perform semantic-level analysis on the data, identify encrypted data and semantically implied sensitive information in the data, determine the identified information as sensitive data, and determine the sensitivity level of the sensitive data.

[0072] Traditional regular expression matching cannot identify implicit sensitive information. This embodiment employs a combined model: RoBERTa (Robustly optimized BERT approach) as a pre-trained language model, providing powerful character vector representations; Bidirectional LSTM (BiLSTM) captures the bidirectional features of the text's context; and CRF (Conditional Random Field) uses label transition probabilities to constrain the sequence labeling results, ensuring the legality of the labels. This model can accurately identify sensitive entities in the text and determine their sensitivity level based on a pre-defined rule base.

[0073] S10242. Construct a de-identification strategy decision tree based on the data usage scenario, user roles, and permission levels.

[0074] The system constructs a decision tree, with the root node representing the data request. Branches are based on factors including: scenario (internal analysis / external sharing), role (administrator / analyst), and data sensitivity level. Leaf nodes represent specific de-identification actions.

[0075] S10243. Based on the desensitization strategy decision tree, automatically select partial masking, field replacement, data generalization or differential privacy algorithm to perform dynamic desensitization processing on sensitive data.

[0076] Based on the output of the decision tree, the system dynamically performs de-identification. For example, partial masking replaces the middle digits of an ID number with asterisks; field replacement replaces real names with pseudonyms; and data generalization generalizes specific values ​​to ranges. For statistical queries, a differential privacy algorithm is used. The principle of this algorithm is that, for any two datasets that differ by only one record, the probability ratio of the algorithm producing a specific output on these datasets must be limited to an exponential range determined by the privacy budget (usually also including a very small relaxation term). In practice, the system adds random noise following a Laplace or Gaussian distribution to the query results, making it impossible for attackers to infer individual information while ensuring the usability of the statistical results.

[0077] S10244, Attribute-Based Access Control (ABAC) model, binds anonymized data to user attributes to achieve dynamic authorization and access auditing.

[0078] The ABAC model dynamically calculates access permissions based on user attributes, resource attributes, and environment attributes, ensuring that only compliant users can see the anonymized data and recording all access behaviors for auditing purposes.

[0079] S103. Store the treated high-quality data in the lake warehouse integrated storage architecture, build a full-text search index and data version snapshot, and provide the high-quality data to users through the application interface layer.

[0080] S1031. Store the treated high-quality data in the lake warehouse integrated storage architecture to build a full-text search index and data version snapshot.

[0081] The architecture utilizes a unified lakeware architecture combining Hudi and ClickHouse. Hudi supports incremental data updates and version management (Time Travel), while ClickHouse provides exceptional columnar query performance. Additionally, Elasticsearch is used to build a full-text index, enabling rapid retrieval of unstructured text.

[0082] S1032. Deploy a natural language processing gateway, integrate it into the ChatGLM model, and receive users' natural language query requests.

[0083] The application interface layer integrates the ChatGLM large language model as an intelligent gateway. Users can directly input natural language query requests.

[0084] S1033. Convert natural language query requests into standardized structured query language SQL or application programming interface (API) call instructions, and send them to the lake warehouse integrated storage architecture for data retrieval.

[0085] The ChatGLM model, after fine-tuning (Text-to-SQL task), can understand the user's intent and translate it into accurate SQL statements or API call parameters.

[0086] S1034. Obtain the search results and generate reports or real-time dashboards using the pre-built visualization component library, and provide feedback to the user.

[0087] S1035. Use the service monitoring module to collect the call frequency, response time and error rate of the interface in real time, encapsulate them into service performance data and push them to the closed-loop optimization and analysis process.

[0088] S104. Real-time collection of service call data from the application interface layer and event data during the governance process; closed-loop optimization analysis based on rule engine and reinforcement learning algorithm; generation of optimization strategy; incremental training of the model in the generative AI core layer; and obtaining the updated data processing model.

[0089] This is a key step in achieving the system's "self-evolution".

[0090] S1041. Use a rule engine to automate the handling of routine abnormal events.

[0091] For known events with fixed processing logic (such as disk space warnings), the rule engine will match preset rules and automatically trigger log cleanup or retry operations.

[0092] S1042. When encountering an unplanned emergency, activate the intelligent contingency plan generation module and use the Case-Based Reasoning (CBR) algorithm to retrieve similar cases from the historical event database.

[0093] The CBR algorithm calculates the similarity between the current event feature vector and the feature vectors in the historical case library (e.g., calculating Euclidean distance), retrieves the most similar historical cases and their solutions, and uses them as a reference benchmark.

[0094] S1043. Combining the standard contingency plan library, the retrieved cases are iteratively optimized using reinforcement learning (RL) algorithms to generate the optimal handling plan, and tasks are allocated through service mesh technology.

[0095] For new problems, simply copying past solutions may be ineffective. The system introduces a reinforcement learning agent for dynamic programming.

[0096] S10431. Construct a reinforcement learning environment, defining the state space as the current event characteristics and system resource status, and the action space as the set of handling measures. The state space includes the current alarm type, system load, network bandwidth, etc.; the action space includes operations such as restarting services, expanding resources, and isolating nodes.

[0097] S10432. Define a reward function, which is constructed based on processing time, resource consumption, and the degree of business recovery. The construction principle of the reward function is: inversely proportional to the expected recovery time (or directly proportional to the recovery speed), inversely proportional to the computational resource cost consumed in executing the action (i.e., the higher the cost, the lower the reward), and directly proportional to the quality score after business recovery. Through this weighted combination, the agent is guided to find the solution with the lowest cost, fastest recovery, and best quality.

[0098] S10433. A reinforcement learning agent performs trial-and-error learning in a reinforcement learning environment. The policy network is updated by maximizing cumulative rewards until the policy converges, and the optimal action sequence is output as the optimal solution. The agent uses Q-learning or Proximal Policy Optimization (PPO) algorithms to perform multiple rounds of trial and error in a simulated environment. In each round, the agent observes the state, selects an action, receives a reward, and transitions to a new state. By continuously updating the policy network parameters, the agent eventually learns which action sequence to choose in a specific state to maximize long-term cumulative rewards. The converged policy is the optimal solution.

[0099] S1044. After the incident is handled, a debriefing report is generated and feature data is extracted. The feature data is then fed back into the metadata management, quality inspection, standard management and security protection models using an incremental training algorithm to update the model parameters and rule base.

[0100] Finally, the system uses the processing flow and feedback of this event as new training samples. Through online learning or incremental learning algorithms, the GenAI model in step S102 is fine-tuned, and its parameters are updated, thereby achieving closed-loop optimization and continuous capability improvement of the entire data processing system.

[0101] As can be seen from the detailed description of the above embodiments, the present invention utilizes deep learning and GenAI technology to reconstruct all aspects of data governance and introduces reinforcement learning to achieve self-evolution, effectively solving many pain points in the prior art and having extremely high industrial application value.

[0102] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

[0103] Secondly, embodiments of the present invention also provide a data query system based on semantic recognition, referring to... Figure 2 The diagram illustrates a functional block diagram of a semantic recognition-based data query system 300 according to the present invention. This system may include the following modules: The data acquisition and preprocessing module 301 is used to access multi-source heterogeneous raw data through a distributed data acquisition framework, and to use a pre-built deep learning classification model and data cleaning algorithm library to perform format recognition and preprocessing on the raw data to obtain standardized preprocessed data. The generative AI core processing module 302 is used to input preprocessed data into the generative AI core layer, and utilize multiple pre-trained and fine-tuned generative artificial intelligence models to perform intelligent metadata management, intelligent data quality detection and repair, automated data standard fusion, and data security and privacy protection processing in parallel to obtain high-quality data after governance. The storage and service management module 303 is used to store the high-quality data after governance into the lake warehouse integrated storage architecture, build full-text search indexes and data version snapshots, and provide high-quality data to users through the application interface layer; The closed-loop optimization module 304 is used to collect service call data and event data in the application interface layer in real time and to perform closed-loop optimization analysis based on the rule engine and reinforcement learning algorithm, generate optimization strategies, and incrementally train the model in the generative AI core layer to obtain the updated data processing model.

[0104] A third aspect of the present invention provides an electronic device, please refer to [link to relevant documentation]. Figure 3The system includes a memory and a processor connected in series and in communication. The memory stores computer programs, and the processor reads the computer programs and executes a model-based data processing method as described in the first aspect of the embodiment. Specifically, the memory may include, but is not limited to, random access memory (RAM), read-only memory (ROM), flash memory, first-in-first-out (FIFO) memory, and / or last-in-last-out (FILO) memory, etc. Since this invention relates to the training and inference of generative AI models (such as GAN, BERT, ChatGLM), which requires high computing power, the processor may not be limited to high-performance central processing units (CPUs), graphics processing units (GPUs), tensor processors (TPUs), or server-grade processors with integrated NPUs (such as Intel Xeon series, NVIDIA Tesla / Ampere series, etc.) to meet the needs of large-scale data processing and deep learning model computation. The working process, working details, and technical effects of the electronic device provided in the third aspect of this embodiment can be found in the first aspect of the embodiment, and will not be repeated here.

[0105] This fourth aspect of the embodiment provides a computer-readable storage medium storing instructions containing a model-learning-based data processing method according to the first aspect of the embodiment. Specifically, the computer-readable storage medium stores instructions that, when executed on a computer, perform the model-learning-based data processing method as described in the first aspect. The computer-readable storage medium refers to a data storage medium, which may include, but is not limited to, floppy disks, optical disks, hard disks, flash memory, USB flash drives, and / or Memory Sticks. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The working process, details, and technical effects of the computer-readable storage medium provided in this fourth aspect of the embodiment can be found in the first aspect of the embodiment, and will not be repeated here.

[0106] The fifth aspect of this embodiment provides a computer program product containing instructions that, when executed on a computer, cause the computer to perform a model-based data processing method as described in the first aspect of this embodiment. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices.

[0107] The embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0108] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., including several instructions to cause a computer or programmable device to execute the methods of various embodiments or some parts of embodiments.

[0109] Finally, it should be noted that although embodiments of the present invention have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the present invention, the scope of which is defined by the claims and their equivalents.

Claims

1. A data processing method based on model learning, characterized in that, The method includes: The system accesses multi-source heterogeneous raw data through a distributed data acquisition framework, and uses a pre-built deep learning classification model and data cleaning algorithm library to perform format recognition and preprocessing on the raw data to obtain standardized preprocessed data. The preprocessed data is input into the generative AI core layer, and multiple pre-trained and fine-tuned generative artificial intelligence models are used to perform intelligent metadata management, intelligent data quality detection and repair, automated data standard fusion, and data security and privacy protection processing in parallel to obtain high-quality data after governance. The high-quality data after treatment is stored in the lake warehouse integrated storage architecture, a full-text search index and data version snapshot are constructed, and the high-quality data is converted into data services and provided to users through the application interface layer; The system collects service call data and event data from the application interface layer in real time, performs closed-loop optimization analysis based on the rule engine and reinforcement learning algorithm, generates optimization strategies, and incrementally trains the model in the generative AI core layer to obtain an updated data processing model.

2. The data processing method based on model learning according to claim 1, characterized in that, The process involves using a pre-built deep learning classification model and data cleaning algorithm library to perform format recognition and preprocessing on the original data, resulting in standardized preprocessed data, including: A deep learning classification model combining a convolutional neural network (CNN) and a long short-term memory network (LSTM) is used to extract and fuse the file header features and content sequence features of the original data to identify the data format of the original data. Based on the identified data format, the corresponding parsing strategy is invoked to convert the raw data into a unified data stream; The SimHash algorithm is used to calculate the fingerprint features of the data stream and perform deduplication. At the same time, the numerical data is preliminarily screened and filtered for outliers based on the 3σ principle to obtain standardized preprocessed data. The preprocessed data is pushed to the generative AI core layer via a message queue.

3. The data processing method based on model learning according to claim 1, characterized in that, The method of utilizing multiple pre-trained and fine-tuned generative artificial intelligence models to perform intelligent metadata management in parallel includes: The ANTLR4 parser is used to perform syntax tree analysis on the scripts in the preprocessed data to extract physical metadata of table structure, field names, and data types; The pre-trained natural language processing model ERNIE 3.0 is invoked, and combined with a pre-built enterprise business dictionary, semantic mapping analysis is performed on the extracted physical metadata to generate standardized metadata tags containing business meaning and data source. Collect read and write operations and interface call relationships from data processing logs, and use graph neural networks (GNNs) to construct a directed graph of data nodes and processing nodes to generate a data lineage graph. When a change in the data link is detected, the data lineage map is refreshed within a preset time window using an incremental update algorithm.

4. The data processing method based on model learning according to claim 1, characterized in that, Performing the aforementioned intelligent data quality detection and repair includes: A dynamic baseline model is built based on Generative Adversarial Network (GAN). The generator of DCGAN architecture generates simulated normal data based on historical normal data, and the discriminator of CNN architecture distinguishes between real normal data and simulated normal data until the loss function converges, thus determining the normal data baseline. Principal Component Analysis (PCA) algorithm is used to reduce the dimensionality of real-time incoming data and extract key features, and the cosine similarity between the key features and the normal data benchmark is calculated. When the cosine similarity is lower than a preset threshold, the data is determined to be abnormal and an early warning is triggered. The domain knowledge graph is called to perform contextual association analysis on the abnormal data in order to locate the root cause of the abnormality. Repair rules are generated based on entity relationships in the domain knowledge graph to automatically complete or correct simple anomalies; for complex anomalies, a visual report containing repair suggestions is generated.

5. The data processing method based on model learning according to claim 1, characterized in that, Performing the automated fusion of the data standards includes: Using the Few-Shot learning framework, enterprise industry attributes, business scope, and historical data samples are input into a pre-trained BERT model for semantic encoding, generating a draft data standard that includes data format, encoding rules, and value range. The Sentence-BERT model is used to convert existing data standards from different departments into semantic vectors, calculate the semantic similarity of cross-departmental standards, identify standard differences, and generate a difference comparison matrix. Based on the genetic algorithm, a fusion scheme is searched for the identified standard differences. The objective function is to maximize data consistency and minimize business impact. The optimal data standard fusion strategy is then output through iterative optimization. Based on the optimal data standard fusion strategy, the preprocessed data is standardized, mapped, and transformed.

6. The data processing method based on model learning according to claim 1, characterized in that, Performing the data security and privacy protection processing includes: The RoBERTa-BiLSTM-CRF deep learning model is used to perform semantic-level analysis on the data, identify encrypted data and semantically implied sensitive information in the data, determine the identified information as sensitive data, and determine the sensitivity level of the sensitive data. Construct a de-identification strategy decision tree based on the data usage scenario, user roles, and permission levels; Based on the aforementioned desensitization strategy decision tree, the system automatically selects partial masking, field replacement, data generalization, or differential privacy algorithms to dynamically desensitize the sensitive data. The attribute-based access control ABAC model binds anonymized data to user attributes to achieve dynamic authorization and access auditing.

7. The data processing method based on model learning according to claim 1, characterized in that, The application interface layer converts the high-quality data into data services and provides them to users, including: Deploy a natural language processing gateway and integrate it into the ChatGLM model to receive users' natural language query requests; The natural language query request is converted into a standardized structured query language (SQL) or application programming interface (API) call command, and then sent to the lake warehouse integrated storage architecture for data retrieval; Obtain search results and generate reports or real-time dashboards using a pre-built library of visualization components to provide feedback to users; The service monitoring module collects API call frequency, response time, and error rate metrics in real time, encapsulates them into service performance data, and pushes them into a closed-loop optimization and analysis process.

8. The data processing method based on model learning according to claim 1, characterized in that, The closed-loop optimization analysis based on the rule engine and reinforcement learning algorithm, generating optimization strategies, and incrementally training the model in the generative AI core layer includes: Use a rule engine to automate the handling of common abnormal events; When encountering unforeseen emergencies, the intelligent contingency plan generation module is activated, and the case reasoning CBR algorithm is used to retrieve similar cases from the historical event database. By combining a standard contingency plan library, reinforcement learning (RL) algorithms are used to iteratively optimize the retrieved cases, generate the optimal response plan, and allocate tasks through service mesh technology. After the incident is handled, a debriefing report is generated and feature data is extracted. The feature data is then fed back into the models for metadata management, quality inspection, standard management, and security protection using an incremental training algorithm to update the model parameters and rule base.

9. The data processing method based on model learning according to claim 8, characterized in that, The optimization process of the reinforcement learning (RL) algorithm includes: Construct a reinforcement learning environment, defining the state space as the current event characteristics and system resource state, and the action space as the set of disposal measures; Define a reward function, which is constructed based on the processing time, resource consumption, and business recovery level; The reinforcement learning agent performs trial and error learning in the reinforcement learning environment, updates the policy network by maximizing the cumulative reward until the policy converges, and outputs the optimal treatment sequence as the optimal treatment plan.

10. A data processing system based on model learning, characterized in that, The system includes: The data acquisition and preprocessing module is used to access multi-source heterogeneous raw data through a distributed data acquisition framework, and to perform format recognition and preprocessing on the raw data using a pre-built deep learning classification model and data cleaning algorithm library to obtain standardized preprocessed data. The generative AI core processing module is used to input the preprocessed data into the generative AI core layer, and utilize multiple pre-trained and fine-tuned generative artificial intelligence models to perform intelligent metadata management, intelligent data quality detection and repair, automated data standard fusion, and data security and privacy protection processing in parallel to obtain high-quality data after governance. The storage and service management module is used to store the high-quality data after governance into the lake warehouse integrated storage architecture, build a full-text search index and data version snapshot, and convert the high-quality data into data services for users through the application interface layer; The closed-loop optimization module is used to collect service call data and event data in the governance process of the application interface layer in real time, perform closed-loop optimization analysis based on the rule engine and reinforcement learning algorithm, generate optimization strategies, and incrementally train the model in the generative AI core layer to obtain the updated data processing model.

Images