Data processing method and device, equipment, storage medium and program product

Abstract

The embodiment of the invention provides a data processing method and device, equipment, a storage medium and a program product, and the method comprises the steps: carrying out the format cleaning of an extracted original data file, and obtaining a to-be-loaded data file; based on the abnormal file name set, determining a verification sequence of the to-be-loaded data file, and verifying the to-be-loaded data file according to a multi-layer rule verification model and an anomaly detection model; and when the verification is passed, loading the to-be-loaded data file into the target database, and when the verification is not passed, performing exception repair on the to-be-loaded data file, loading the repaired to-be-loaded data file into the target database, and updating the exception file name set according to the to-be-loaded data file. The accuracy and the intelligent level of anomaly detection are improved by fusing rule verification and anomaly detection model verification; by caching the abnormal file name, the verification priority of the abnormal file and the similar file is improved, and the anomaly detection efficiency is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, and in particular to a data processing method and device, equipment, storage medium and program product. BACKGROUND

[0002] In the big data era, data has become an important core production factor in the new era. As the core link of data warehouse construction, ETL (Extract-Transform-Load) technology is responsible for extracting data from the source system, cleaning and converting, and loading into the target database. In the big data scenario, due to large data volume and complex calculation, ETL gradually evolves into ELT, that is, first extract and load data, and then convert in the data warehouse.

[0003] The data loading stage refers to loading data in the upstream source system into the target database through various methods, and the accuracy and availability of the loaded data directly affect the downstream data processing and analysis effect. The current rule-based verification method relies on manual rule setting, lacks flexibility and intelligence, has low verification efficiency, and has weak detection ability for novel anomalies. SUMMARY

[0004] The embodiments of the present application provide a data processing method, device, equipment, storage medium and program product to improve the accuracy and availability of loaded data.

[0005] In a first aspect, the embodiments of the present application provide a data processing method, which comprises:

[0006] According to the preprocessing rule, the original data file is format cleaned to obtain a to-be-loaded data file;

[0007] According to the abnormal file name set, the verification order of the to-be-loaded data file is determined;

[0008] Based on the verification order, the to-be-loaded data file is verified according to the multi-layer rule verification model and the anomaly detection model to obtain an abnormal verification result;

[0009] In the case that the abnormal verification result is passed, the to-be-loaded data file is loaded into the target database, and in the case that the abnormal verification result is not passed, the to-be-loaded data file is repaired, the repaired to-be-loaded data file is loaded into the target database, and the abnormal file name set is updated according to the to-be-loaded data file.

[0010] In a possible implementation, according to the abnormal file name set, the verification order of the to-be-loaded data file is determined, which comprises:

[0011] It is judged whether the to-be-loaded data file hits the abnormal file name set;

[0012] In the case of a hit, the check priority of the to-be-loaded data file is adjusted, and the to-be-loaded data file is added to the head of the processing queue. In the case of a miss, the check priority of the to-be-loaded data file remains unchanged.

[0013] In a possible implementation, the to-be-loaded data file is checked according to the multi-layer rule check model and the anomaly detection model to obtain an anomaly check result, including:

[0014] According to the multi-layer rule check model, the to-be-loaded data file is checked from the encoding layer, the character layer, and the metadata layer to obtain a first sub-check result.

[0015] The metadata features of the to-be-loaded data file are extracted, and the metadata features are input into the trained anomaly detection model to obtain a second sub-check result.

[0016] According to the first sub-check result and the second sub-check result, the anomaly check result of the to-be-loaded data file is determined.

[0017] In a possible implementation, according to the multi-layer rule check model, the to-be-loaded data file is checked from the encoding layer, the character layer, and the metadata layer to obtain a first sub-check result, including:

[0018] When checking the to-be-loaded data file from the metadata layer, the to-be-loaded data file is divided into multiple partitions, and the field-level data in each partition is checked to obtain a data check result of each partition.

[0019] According to the data check result of each partition, the check result of the to-be-loaded data file in the metadata layer is determined.

[0020] In a possible implementation, the anomaly file name set is updated according to the to-be-loaded data file, including:

[0021] According to the feature distance between the instance in the to-be-loaded data file and the instance in other data files, the similarity between the to-be-loaded data file and other data files is determined.

[0022] The to-be-loaded data file and other files with a similarity greater than a preset threshold to the to-be-loaded data file are added to the anomaly file name set.

[0023] In a possible implementation, the to-be-loaded data file is loaded into the target database, including:

[0024] The loading mode corresponding to the to-be-loaded data file is determined.

[0025] According to the loading mode, the to-be-loaded data file is loaded into the target database.

[0026] In a second aspect, an embodiment of the present application provides a data processing apparatus, the apparatus comprising:

[0027] a preprocessing module configured to perform format cleaning on the extracted original data file according to a preprocessing rule to obtain a to-be-loaded data file;

[0028] a sequence determination module configured to determine a checking sequence of the to-be-loaded data file according to the set of abnormal file names;

[0029] a data checking module configured to check the to-be-loaded data file based on the checking sequence, according to a multi-layer rule checking model and an anomaly detection model, to obtain an abnormal checking result;

[0030] a data loading module configured to load the to-be-loaded data file into the target database if the abnormal checking result is passed, and perform abnormal repair on the to-be-loaded data file if the abnormal checking result is not passed, load the repaired to-be-loaded data file into the target database, and update the set of abnormal file names according to the to-be-loaded data file.

[0031] In a third aspect, an embodiment of the present application provides a computer device, comprising: a memory, a processor; the memory stores computer execution instructions; the processor executes the computer execution instructions stored in the memory, so that the processor executes the first aspect and / or various possible implementation manners of the first aspect.

[0032] In a fourth aspect, an embodiment of the present application provides a computer readable storage medium, the computer readable storage medium stores computer execution instructions, and the computer execution instructions are executed by the processor to implement the first aspect and / or various possible implementation manners of the first aspect.

[0033] In a fifth aspect, an embodiment of the present application provides a computer program product, comprising a computer program, the computer program is executed by the processor to implement the first aspect and / or various possible implementation manners of the first aspect.

[0034] The data processing method, apparatus, device, storage medium, and program product provided in this application include: cleaning the format of extracted raw data files according to preprocessing rules to obtain data files to be loaded; determining the verification order of the data files to be loaded based on a set of abnormal filenames; verifying the data files to be loaded based on the verification order, according to a multi-layer rule verification model and an anomaly detection model, to obtain an anomaly verification result; loading the data files to be loaded into a target database if the anomaly verification result is successful; and repairing the data files to be loaded and loading the repaired data files into the target database, while updating the set of abnormal filenames based on the data files to be loaded. By integrating rule verification and anomaly detection model verification, the accuracy and intelligence level of anomaly detection are improved; by caching abnormal filenames, the verification priority of abnormal files and similar files is increased, thereby improving anomaly detection efficiency. Attached Figure Description

[0035] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0036] Figure 1 Flowchart of the data processing method provided in this application Figure 1 ;

[0037] Figure 2 A flowchart illustrating the process for determining the verification order provided in this application;

[0038] Figure 3 A schematic diagram illustrating the principle of the anomaly detection model provided in this application;

[0039] Figure 4 Flowchart of the data processing method provided in this application Figure 2 ;

[0040] Figure 5 Flowchart of the data processing method provided in this application Figure 3 ;

[0041] Figure 6 A schematic diagram of the data processing apparatus provided in this application;

[0042] Figure 7 A schematic diagram of the structure of the computer device provided in this application.

[0043] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0044] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0045] In the description of this application, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this application, "multiple" means two or more, unless otherwise explicitly specified.

[0046] DDL: Data Definition Language.

[0047] In the era of big data, data has become a crucial core production factor. In the financial sector, for example, data plays a vital role in risk management, decision support, daily operations, and customer experience. ETL (Extract-Transform-Load) technology, as a core component of data warehouse construction, is responsible for extracting data from source systems, cleaning and transforming it, and loading it into the target database. In big data scenarios, due to the large volume of data and the complexity of computation, ETL has gradually evolved into ELT (Extract-Load-Transform), which involves first extracting and loading data, and then transforming it within the data warehouse.

[0048] The data loading phase refers to loading data from the upstream source system into the target database using various methods. The accuracy and availability of the loaded data directly affect the downstream data processing and analysis results. Currently, scheduling tools and control scripts are used to implement data loading. The upstream source system distributes data in the form of files, and the cluster then loads the distributed data into the data lake or data warehouse. In this process, there is a lack of quality control over the data files distributed from the upstream. If the data has quality issues, the loading process often relies on the database's own characteristics to check for data format problems, leading to the following issues in the data validation and loading process:

[0049] Data quality issues: When directly loading data into a Hive table, for DECIMAL type data, if the precision of the data inserted into the Hive table exceeds the definition, it will not directly report an error, but will round it to the specified precision, which may lead to data distortion and failure to provide timely feedback to the upstream system.

[0050] Low verification efficiency: Currently, verification is based on data type verification rules, which rely on manual formulation and are fixed in the program, lacking flexibility and intelligence, resulting in low verification efficiency and weak ability to detect novel anomalies.

[0051] To address the aforementioned technical problems, embodiments of this application provide a data processing method for performing multi-faceted verification and processing on data files distributed by an upstream source system, thereby improving the accuracy and usability of loaded data. For example... Figure 1 As shown, in one embodiment, the data processing method includes:

[0052] Step 101: Clean the format of the extracted raw data file according to the preprocessing rules to obtain the data file to be loaded;

[0053] The raw data file is the output of the data extraction phase, including all data extracted from the upstream source system. The raw data file can be actively pulled from the upstream source system by the extraction process, such as by actively connecting to the source of the upstream source system through the ELT tool, such as by directly connecting to the production database or calling application programming interfaces (APIs). Alternatively, it can be distributed by the upstream source system. Specifically, the upstream source system generates a file from the data and pushes or distributes it to a pre-agreed location. The ELT process then locates and extracts this file from that pre-agreed location, thus obtaining the raw data file.

[0054] Format cleaning refers to the process of adjusting and correcting the content, structure, and format of a data file to ensure it meets the target system's expectations and requirements for the data format, thus preventing errors and interruptions during the loading phase due to format issues. Format cleaning is performed sequentially before loading, addressing syntax issues rather than semantic ones.

[0055] Preprocessing rules are rules set for format cleaning. Format cleaning includes removing spaces, converting date / time formats, handling delimiters, unifying character encoding, and handling null or missing values. The original data file is processed using the preprocessing rules at the system configuration layer to obtain the data file to be loaded.

[0056] Step 102: Determine the verification order of the data files to be loaded based on the set of abnormal file names;

[0057] The abnormal filename set refers to a collection of data files that frequently or recently resulted in errors. This set stores the filenames of all frequently encountered error files and can be stored in Redis (Remote Dictionary Server) to accelerate error detection. The abnormal filename set can be manually configured or automatically updated based on the error verification results of the data files to be loaded.

[0058] The verification order refers to the order in which multiple data files are loaded when there are multiple data files to be loaded. One loading job loads one data file, and queuing may occur when additional loading jobs are added. For each data file to be loaded, before verification, a preliminary verification is performed based on the set of abnormal filenames to determine if its filename exists in the set of abnormal filenames. If it does, it is verified first to speed up the discovery of abnormal files.

[0059] Specifically, in one embodiment, the verification order of the data files to be loaded is determined based on the set of abnormal filenames, including:

[0060] Determine if the data file to be loaded matches the set of abnormal filenames;

[0061] If a check is successful, adjust the verification priority of the data file to be loaded and add it to the head of the processing queue. If a check is unsuccessful, keep the verification priority of the data file to be loaded unchanged.

[0062] A newly added loading job is treated as a new task. Its corresponding data file to be loaded is searched in the set of abnormal filenames. If a match is found, the data file is highly likely to be an abnormal file, and its verification priority is raised to the highest level, inserted at the head of the verification queue for priority verification. Simultaneously, data files similar to the one to be loaded can be added to the set of abnormal filenames. If no match is found, the file is added to the verification queue and processed as a normal element in the queue.

[0063] It should be noted that when there are multiple verification queues, the aforementioned inserted verification queue refers to the verification queue that has already been determined through routing. For example... Figure 2 As shown, multiple abnormal filenames [J1, J2, J3, J4, J5, J8, ...] are stored in Redis. For the data file J1 to be loaded, the system first checks whether it matches the target file in Redis. If it does, a target queue is determined from Queue1, Queue2, and Queue3, and J1 is placed at the head of the target queue for priority processing. Otherwise, it is queued normally in Queue1, Queue2, and Queue3.

[0064] Furthermore, during the match detection phase, the set of abnormal filenames can be updated based on the match results. Specifically, each filename in the abnormal filename set has a corresponding expiration time. If the data file to be loaded matches the match, the expiration time of that filename is extended; otherwise, the filename is removed from the abnormal filename set after the expiration time has expired.

[0065] Step 103: Based on the verification order, the data file to be loaded is verified according to the multi-level rule verification model and the anomaly detection model to obtain the anomaly verification result.

[0066] A multi-level rule-based validation model refers to validating data files at multiple levels; it is essentially a rule-based validation process. These multiple levels of a data file relate to the accuracy and reliability of the data within it. By validating the data file to be loaded using a multi-level rule-based validation model, and based on the validation results at multiple levels, the first sub-validation result of the multi-level rule-based validation model for the data file to be loaded is determined. Based on the constructed multi-level rule-based validation model, basic anomaly detection for data files can be covered.

[0067] Anomaly detection models are built upon deep learning models and are used to detect novel anomaly types. Given the abundance of normal samples and the scarcity of anomalous samples in data files, appropriate deep learning models are selected for novel anomaly detection. For example, a Generative Adversarial Network (GAN) can be used to construct an anomaly detection model to supplement the multi-layer rule verification model. Specifically, the loss function of the GAN model is as follows:

[0068] ;

[0069] Generator and discriminator In a zero-sum game, a Nash equilibrium is eventually reached. and Representing the true data distribution and noise variables respectively The prior distribution is used to solve the generator separately. and discriminator For details on calculating the loss function (loss), please refer to [link / reference]. Figure 3 .

[0070] Before using GANs for anomaly file detection, model training is required. The training process includes: First, collecting metadata about the data files, including features related to the data files and anomalies, such as: data file name, source system name, subsystem name, data arrival time, data file extension, data file size before distribution, actual data file size, file encoding type, data file arrival time, presence of signal files, and whether errors have occurred within the last 15 days. Next, data preprocessing is performed, converting the raw data to numerical values, imputing missing values, and normalizing the feature data to obtain a metadata feature dataset. Model training and testing are then conducted using the collected metadata feature dataset to identify the boundaries of positive example features. Finally, the trained model is deployed to perform anomaly detection on the data files to be loaded, obtaining a second sub-verification result.

[0071] When validating a data file to be loaded based on a multi-level rule-based validation model and an anomaly detection model, the data file can be input into both models simultaneously. The first and second sub-validation results are then integrated to obtain the anomaly validation result. The anomaly validation result is considered successful only if both the first and second sub-validation results pass; otherwise, it is considered unsuccessful. In some embodiments, the multi-level rule-based validation model and the anomaly detection model are fused to obtain a multi-RGAN model for anomaly data file validation. After the data file arrives at the target system, rule validation and GAN anomaly detection are performed simultaneously. Data loading only proceeds if both models validate successfully.

[0072] In addition, the data file to be loaded can be sequentially input into the multi-level rule validation model and the anomaly detection model for two validations. If the first validation fails, the anomaly detection result can be directly determined as failed. If the first validation passes, a second validation is performed to determine the anomaly detection result. However, in normal circumstances, since the multi-level rule validation model needs to validate the data from multiple levels, which is relatively resource-intensive, the anomaly detection model is used for validation first.

[0073] For anomaly detection models, model update trigger conditions can be set, such as an anomaly detection accuracy drop threshold and the accumulation of new anomaly samples. New samples are periodically extracted from historical validation logs to incrementally update the GAN model. An online learning framework is adopted, allowing the anomaly detection model to fine-tune its parameters after each validation with new data, adapting to changes in data distribution. The trained models are then stored in versions, and the optimal version is selected for validation based on the current data characteristics.

[0074] Step 104: If the exception verification result is successful, load the data file to be loaded into the target database. If the exception verification result is unsuccessful, perform exception repair on the data file to be loaded, load the repaired data file into the target database, and update the exception file name set according to the data file to be loaded.

[0075] The data file to be loaded is verified based on a multi-layer rule model and anomaly detection model. The data file can be verified from multiple levels. If the anomaly verification result fails and meets the self-healing conditions, the self-healing module is entered to attempt self-healing to repair the abnormal data before reloading. If the self-healing rules are not met, an error is directly reported and no further loading operation is performed, which can reduce manual intervention.

[0076] The automatic repair of the data file to be loaded mainly involves removing line breaks, special characters, etc. from the data file to be loaded. The specific scenarios that can be removed need to be negotiated with the upstream source system. Then, a new data file to be loaded is used for reloading, so that the upstream source system does not need to push data again, thus achieving self-healing of the problem.

[0077] Furthermore, to ensure the atomicity and rollback of the repair operation, a transaction mechanism can be introduced during the repair process. The original data files are backed up before the repair operation, and if the repair fails, the system rolls back to the original state. After the repair, a multi-layered rule validation model and anomaly detection model are used again to ensure that the repaired data conforms to the rules; if anomalies still exist, a manual review process is triggered. The repair rule whitelist is defined based on the data source, such as allowing only the removal of specific special characters to avoid accidental operations.

[0078] If the exception verification result is successful, the data file to be loaded will be loaded into the target database. This can be done by loading the data file into a Hive table, a Gaussian database, or other types of databases; different loading targets can be configured.

[0079] Specifically, in one embodiment, loading the data file to be loaded into the target database includes:

[0080] Determine the loading method for the data file to be loaded;

[0081] Depending on the loading method, the data file to be loaded is loaded into the target database.

[0082] The loading methods include Beeline, Spark SQL, Spark functions and methods, and GAUSS DB tables.

[0083] In one embodiment, the method further includes: monitoring, recording, and persisting key information about specific job execution during the loading process. For example, from a business perspective, it may be desirable to have a comprehensive understanding of the entire data loading process into the lake, such as which jobs encountered errors, the content of the errors, the size of each loaded data file, which jobs had the longest execution time, and which jobs were delayed by late upstream deployments, thus slowing down the entire chain's execution time. This is achieved by collecting relevant key metrics during job execution and persisting the collected information to a table after job completion. This layer primarily stores information such as job start time, job end time, job name, statistical date, number of data rows loaded into the Hive table, ETL server domain name where the job was executed, data file size, data file name, signal file name, data file deployment time, and job error content in the database for subsequent analysis and research. If a loading error occurs, the corresponding job manager will be alerted via SMS and email; execution information will be printed, and execution logs will be recorded.

[0084] In the method provided in the above embodiments, the extracted raw data files are format-cleaned according to preprocessing rules to obtain data files to be loaded; the verification order of the data files to be loaded is determined according to the abnormal file name set; based on the verification order, the data files to be loaded are verified according to a multi-layer rule verification model and an anomaly detection model to obtain an anomaly verification result; if the anomaly verification result is successful, the data files to be loaded are loaded into the target database; if the anomaly verification result is unsuccessful, the data files to be loaded are repaired, the repaired data files are loaded into the target database, and the abnormal file name set is updated according to the data files to be loaded. By integrating rule verification and anomaly detection model verification, the accuracy and intelligence level of anomaly detection are improved; by caching abnormal file names, the verification priority of abnormal files and similar files is increased, thereby improving anomaly detection efficiency.

[0085] In one embodiment, such as Figure 4 As shown, based on the multi-layer rule verification model and the anomaly detection model, the data file to be loaded is verified, and the anomaly verification results are obtained, including:

[0086] Step 401: According to the multi-layer rule verification model, the data file to be loaded is verified from the encoding layer, character layer and metadata layer to obtain the first sub-verification result;

[0087] Step 402: Extract the metadata features of the data file to be loaded, and input the metadata features into the trained anomaly detection model to obtain the second sub-verification result;

[0088] Step 403: Determine the abnormal verification result of the data file to be loaded based on the first sub-verification result and the second sub-verification result.

[0089] The multi-layer rule-based validation model divides rule-based anomaly detection into three layers:

[0090] The first layer is the coding layer: it is used to detect whether there are abnormal encodings or garbled characters in the data file to be loaded. Abnormal encodings here are relative to the pre-defined constraints. For example, data files issued under general constraints are generally encoded in UT8-8; however, some data files, although they are UTF-8, may have garbled characters at the end of the fields due to issues with export parameters set by the upstream source system, such as truncation according to fixed bytes.

[0091] The second layer is the character layer: it is mainly used to detect special invisible ASCII characters (control characters) that may cause loading errors in the data file to be loaded, as well as other special characters that may cause loading errors. The character layer refers to each character in the data file. If it contains the invisible ASCII characters ^N and ^M, it will cause misalignment when reading the data file and will report an error during loading.

[0092] The third layer is the metadata layer. This layer uses the metadata information of the target table corresponding to the data file to be loaded to validate the data. In some embodiments, the following information is mainly validated: whether the number of data columns in the data file matches the number of fields in the DDL table definition; if they do not match, an error will be reported. Whether the data in the data file conforms to the DDL table definition. The data in the data file does not have a data type; it is simply structured data similar to a CSV file. The field definitions use DDL tables, where the data definitions are negotiated with the upstream source system. The data types need to be validated in conjunction with the DDL definition table in the target system.

[0093] The first sub-verification result is determined based on the above three levels of verification. Specifically, verification can be performed layer by layer in the order of encoding layer, character layer, and metadata layer. The first sub-verification result is considered passed only if the verification result of each layer passes. If the verification result of one layer fails, the first sub-verification result can be directly determined as failed, or subsequent layers of verification can be continued to locate abnormal data at different levels.

[0094] In one embodiment, the data file to be loaded is validated according to a multi-layer rule validation model, from the encoding layer, character layer, and metadata layer, to obtain a first sub-validation result, including:

[0095] When validating the data file to be loaded at the metadata layer, the data file to be loaded is divided into multiple partitions, and the field-level data in each partition is validated to obtain the data validation result of each partition.

[0096] Based on the data verification results of each partition, determine the verification results of the data file to be loaded at the metadata layer.

[0097] Since field-level validation using metadata involves a large amount of computation, in order to improve validation efficiency and leverage the computing power of the distributed cluster, the data file to be loaded is divided into multiple partitions. Field-level data in each partition is validated to obtain the data validation result for each partition. Based on the data validation result for each partition, the validation result of the data file to be loaded at the metadata layer is determined.

[0098] Specifically, for example, the `foreach Partition` operator in Spark can be used to divide the data file into different partitions and validate the data in each partition. In some embodiments, a dynamic partitioning strategy can be adopted: dynamically adjusting the number of partitions based on the data file size, and using `Range Partitioning` or `Hash Partitioning` to optimize data distribution. Load balancing mechanism: Introducing dynamic resource allocation into Spark jobs, adjusting the number of Executors in real time based on the task queue. Partition merging and splitting: Splitting very large partitions; if the data volume of a single partition exceeds a threshold, merging smaller partitions to balance the computational load.

[0099] The method provided in the above embodiments addresses the core pain points of traditional ETL technology in terms of data quality, verification efficiency, intelligent anomaly detection, and operational costs by integrating rule verification and anomaly detection models, introducing a Redis caching acceleration mechanism, and constructing automated preprocessing and self-healing processes. This provides an efficient, intelligent, and reliable solution for source-layer data verification and loading in big data scenarios. Leveraging Spark's distributed computing capabilities, metadata verification tasks are processed in shards, improving verification speed.

[0100] In one embodiment, such as Figure 5 As shown, based on the data file to be loaded, update the set of abnormal filenames, including:

[0101] Step 501: Determine the similarity between the data file to be loaded and other data files based on the feature distance between instances in the data file to be loaded and instances in other data files;

[0102] Step 502: Add the data file to be loaded, as well as other files whose similarity to the data file to be loaded is greater than a preset threshold, to the abnormal file name set.

[0103] The data file contains multiple instances. The similarity between two data files is determined based on the feature distance between the instances. If the similarity between other data files and the data file to be loaded is greater than a preset threshold, the other data files are considered to be similar to the data file to be loaded, and the filenames of both the data file to be loaded and the data file to be loaded are added to the abnormal filename set.

[0104] In some embodiments, similar files can be filtered by calculating the Euclidean distance between files based on filename features such as timestamps and subsystem names.

[0105] The method provided in the above embodiments accelerates the discovery and handling of anomalies by adding the abnormal data files to be loaded and the filenames of other similar data files to the abnormal filename set, dynamically updating the abnormal filename set, and expanding it to similar files.

[0106] In conjunction with the above embodiments, a data loading system is provided for implementing data loading. The entire loading and verification process includes:

[0107] The input layer stores the raw data files of the upstream source system. It contains data that is closest to the source system and will be cleaned up after a period of time to save resources.

[0108] The preprocessing layer preprocesses the raw data file according to the preprocessing rules in the configuration layer before loading, and resolves possible errors in advance (pre-processing). This includes removing leading and trailing spaces, converting date formats, delimiters, etc., to obtain the data file to be loaded.

[0109] The validation layer validates the data file to be loaded. It uses a multi-RGAN model to validate the data file at three levels: encoding, characters, and metadata, combined with a single-class anomaly detection model. If there is a problem during validation and the self-healing condition is met, the file enters the self-healing module to attempt self-healing before reloading. If the self-healing rule is not met, an error is reported directly, and further loading operations are not performed. If the validation passes, the loading layer is entered.

[0110] In the loading layer, due to the large number of loading jobs, queuing often occurs. To more efficiently verify problematic data, Redis is used to store the job names corresponding to the abnormal data files. During job scheduling, problematic jobs are checked first, speeding up the exposure of abnormal data files. The loading layer can select different loading methods to load data into the target based on different loading characteristics and actual needs.

[0111] Self-healing layer: Problem data files self-heal. The automatic repair of problem data files mainly involves removing line breaks, special characters, etc. from the problem data files (the specific scenarios that can be removed need to be negotiated with the upstream source system), and then reloading with a new data file. This way, the upstream system does not need to push data again, thus achieving self-healing of the problem.

[0112] Service Layer: The system also checks the loading execution logs and executes corresponding self-healing steps based on different error keywords, reloading the processed data files to reduce operational burden. After loading is complete, regardless of whether an error occurs, the service layer persists relevant information about the loading process for subsequent operational analysis. If a loading error occurs, the system will also notify the relevant personnel via SMS, email, or other communication tools based on pre-reserved information.

[0113] In the above embodiments, regarding the discovery of abnormal data files, the use of a multi-RGAN model combining a rule-based model and an anomaly detection GAN model improves the automation and intelligence of abnormal data file detection, and also has a good preventive effect on novel anomalies. Regarding the speed of abnormal data file discovery, the use of Spark for distributed metadata-level data validation and the use of Redis to store potential abnormal files, prioritizing validation during queuing, accelerates the discovery of abnormal files. For operations and maintenance personnel, this technical solution, through the setting of a preprocessing layer and a self-healing layer, automates pre- and post-anomaly handling, reducing the workload of operations and maintenance personnel and improving operational efficiency.

[0114] It should be understood that although the steps in the flowcharts of the above embodiments are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the above embodiments may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0115] Based on the same inventive concept, this application also provides a data processing apparatus. The solution provided by this apparatus is similar to the solution described in the above method. Therefore, the specific limitations of one or more data processing apparatus embodiments provided below can be found in the limitations of the data processing method above, and will not be repeated here.

[0116] In one embodiment, such asFigure 6 As shown, the data processing device includes a preprocessing module 601, a sequence determination module 602, a data verification module 603, and a data loading module 604, wherein:

[0117] The preprocessing module 601 is used to clean the format of the extracted raw data file according to the preprocessing rules to obtain the data file to be loaded.

[0118] The sequence determination module 602 is used to determine the verification order of the data files to be loaded based on the set of abnormal file names;

[0119] The data verification module 603 is used to verify the data file to be loaded based on the verification order, according to the multi-level rule verification model and the anomaly detection model, and to obtain the anomaly verification result.

[0120] The data loading module 604 is used to load the data file to be loaded into the target database if the exception verification result is passed, and to repair the exception of the data file to be loaded if the exception verification result is failed, load the repaired data file into the target database, and update the set of exception file names according to the data file to be loaded.

[0121] In one possible implementation, the sequence determination module 602 is specifically used for:

[0122] Determine if the data file to be loaded matches the set of abnormal filenames;

[0123] If a check is successful, adjust the verification priority of the data file to be loaded and add it to the head of the processing queue. If a check is unsuccessful, keep the verification priority of the data file to be loaded unchanged.

[0124] In one possible implementation, the data verification module 603 is specifically used for:

[0125] Based on the multi-layer rule verification model, the data file to be loaded is verified from the encoding layer, character layer and metadata layer to obtain the first sub-verification result;

[0126] Extract the metadata features of the data file to be loaded, input the metadata features into the trained anomaly detection model, and obtain the second sub-verification result;

[0127] Based on the first sub-verification result and the second sub-verification result, determine the abnormal verification result of the data file to be loaded.

[0128] In one possible implementation, the data verification module 603 is specifically used for:

[0129] When validating the data file to be loaded at the metadata layer, the data file to be loaded is divided into multiple partitions, and the field-level data in each partition is validated to obtain the data validation result of each partition.

[0130] Based on the data verification results of each partition, determine the verification results of the data file to be loaded at the metadata layer.

[0131] In one possible implementation, the data loading module 604 is specifically used for:

[0132] The similarity between the data file to be loaded and other data files is determined based on the feature distance between instances in the data file to be loaded and instances in other data files.

[0133] The data file to be loaded, as well as other files whose similarity to the data file to be loaded is greater than a preset threshold, are added to the abnormal filename set.

[0134] In one possible implementation, the data loading module 604 is specifically used for:

[0135] Determine the loading method for the data file to be loaded;

[0136] Depending on the loading method, the data file to be loaded is loaded into the target database.

[0137] Each module in the aforementioned data processing device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the operations corresponding to each module.

[0138] Figure 7 A schematic diagram of the structure of the computer device provided in this application. Figure 7 As shown, the electronic device 70 provided in this embodiment includes at least one processor 701 and a memory 702. Optionally, the device 70 further includes a communication component 703. The processor 701, memory 702, and communication component 703 are connected via a bus 704.

[0139] In a specific implementation, at least one processor 701 executes computer execution instructions stored in memory 702, causing at least one processor 701 to perform the above-described method.

[0140] The specific implementation process of processor 701 can be found in the above method embodiments, and its implementation principle and technical effect are similar. It will not be repeated here.

[0141] In the above embodiments, it should be understood that the processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this invention can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules within the processor.

[0142] The memory may include random access memory (RAM) and may also include non-volatile memory (NVM), such as at least one disk storage device.

[0143] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, the buses shown in the accompanying drawings are not limited to a single bus or a single type of bus.

[0144] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method.

[0145] This application also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the above-described method.

[0146] The aforementioned readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The readable storage medium can be any available medium accessible to a general-purpose or special-purpose computer.

[0147] An exemplary readable storage medium is coupled to a processor, enabling the processor to read information from and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can reside in an Application Specific Integrated Circuit (ASIC). Alternatively, the processor and the readable storage medium can exist as discrete components in the device.

[0148] The division of units is merely a logical functional division; in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices, or units, and may be electrical, mechanical, or other forms.

[0149] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0150] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0151] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0152] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0153] Finally, it should be noted that other embodiments of the invention will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This invention is intended to cover any variations, uses, or adaptations of the invention that follow the general principles of the invention and include common knowledge or customary techniques in the art not disclosed herein, and is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of the invention is limited only by the appended claims.

Claims

1. A data processing method, characterized by, The method comprises: According to the pre-processing rule, the extracted original data file is format cleaned to obtain a to-be-loaded data file; According to the abnormal file name set, the check sequence of the to-be-loaded data file is determined; Based on the check sequence, the to-be-loaded data file is checked according to a multi-layer rule check model and an anomaly detection model to obtain an abnormal check result; In the case that the abnormal check result is passed, the to-be-loaded data file is loaded into the target database; in the case that the abnormal check result is not passed, the to-be-loaded data file is repaired, the repaired to-be-loaded data file is loaded into the target database, and the abnormal file name set is updated according to the to-be-loaded data file.

2. The method of claim 1, wherein, The abnormal file name set is used to determine the check sequence of the to-be-loaded data file, which comprises: Determine whether the to-be-loaded data file hits the abnormal file name set; In the case of hitting, the check priority of the to-be-loaded data file is adjusted, and the to-be-loaded data file is added to the head of the processing queue; in the case of not hitting, the check priority of the to-be-loaded data file remains unchanged.

3. The method of claim 1, wherein, The to-be-loaded data file is checked according to the multi-layer rule check model and the anomaly detection model to obtain an abnormal check result, which comprises: According to the multi-layer rule check model, the to-be-loaded data file is checked from the encoding layer, the character layer and the metadata layer to obtain a first sub-check result; The metadata features of the to-be-loaded data file are extracted, and the metadata features are input into the trained anomaly detection model to obtain a second sub-check result; According to the first sub-check result and the second sub-check result, the abnormal check result of the to-be-loaded data file is determined.

4. The method of claim 3, wherein, According to the multi-layer rule check model, the to-be-loaded data file is checked from the encoding layer, the character layer and the metadata layer to obtain a first sub-check result, which comprises: When checking the to-be-loaded data file from the metadata layer, the to-be-loaded data file is divided into multiple partitions, and the field-level data in each partition is checked to obtain a data check result of each partition; According to the data check result of each partition, the check result of the to-be-loaded data file in the metadata layer is determined.

5. The method of claim 1, wherein, According to the to-be-loaded data file, the abnormal file name set is updated, which comprises: According to the feature distance between instances in the to-be-loaded data file and instances in other data files, the similarity between the to-be-loaded data file and other data files is determined; The to-be-loaded data file and other files with a similarity greater than a preset threshold to the to-be-loaded data file are added to the abnormal file name set.

6. The method of claim 1, wherein, The to-be-loaded data file is loaded into the target database, which comprises: Determine the loading mode corresponding to the to-be-loaded data file; According to the loading mode, the to-be-loaded data file is loaded into the target database.

7. A data processing apparatus, characterized by, The device comprises: A pre-processing module is configured to clean the extracted original data file according to a pre-processing rule to obtain a to-be-loaded data file; The sequence determining module is configured to determine a check sequence of the to-be-loaded data file according to the abnormal file name set; The data check module is configured to check the to-be-loaded data file based on the check sequence, a multi-layer rule check model, and an abnormality detection model, and obtain an abnormality check result; The data loading module is configured to load the to-be-loaded data file into a target database when the abnormality check result is passed, perform abnormality repair on the to-be-loaded data file when the abnormality check result is failed, load the repaired to-be-loaded data file into the target database, and update the abnormal file name set according to the to-be-loaded data file.

8. A computer device, comprising: Comprise: A processor, and a memory connected with the processor in communication; The memory stores computer execution instructions; The processor executes the computer execution instructions stored in the memory to implement the method of any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer readable storage medium stores computer execution instructions, and the computer execution instructions are executed by the processor to implement the method of any one of claims 1 to 6.

10. A computer program product, characterised in that, The computer program is executed by the processor to implement the method of any one of claims 1 to 6.

Images