Data processing method and device

Abstract

The invention provides a data processing method and device, is applied to the technical field of data processing, and is used for improving the efficiency and accuracy of data synchronization. The method comprises the steps of receiving a data synchronization instruction and information of at least one database; in response to the data synchronization instruction, generating at least one flink task; each flink task in the at least one flink task comprises a data synchronization rule corresponding to the preset requirement, and each flink task is used for performing data synchronization processing on a corresponding database according to the data synchronization rule; submitting the at least one flink task to a preset session for parallel processing, and obtaining a processing result of the at least one flink task; and calculating a field abnormal value of the synchronous data corresponding to each flink task, and executing an alarm on the flink task of which the field abnormal value of the synchronous data is greater than a preset threshold value.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a data processing method and device. Background Art

[0002] As business complexity increases, the types of data development channel sources in enterprises also increase. Usually, a series of complex components are needed to synchronize data from various data sources. The development process is cumbersome and it is difficult to process and convert data efficiently, which in turn affects the timeliness and accuracy of data-driven decision-making. Therefore, efficient synchronization of multi-source data has become a major challenge.

[0003] Therefore, how to improve the efficiency and accuracy of data synchronization has become an urgent problem to be solved. Summary of the invention

[0004] The present application provides a data processing method and device for improving the efficiency and accuracy of data synchronization.

[0005] In a first aspect, an embodiment of the present application provides a data processing method, which is applied to any electronic device with processing capabilities, and the method includes:

[0006] Receiving a data synchronization instruction and information of at least one database; the data synchronization instruction is used to instruct to perform data synchronization processing on the data of at least one database according to preset requirements;

[0007] In response to the data synchronization instruction, at least one flink task is generated; each flink task in the at least one flink task includes a data synchronization rule corresponding to the preset requirement, and each flink task in the at least one flink task is used to perform data synchronization processing on a corresponding database in the at least one database according to the data synchronization rule; the data synchronization rule in each flink task is different;

[0008] Submit at least one flink task to a preset session for parallel processing, and obtain the processing result of at least one flink task; the processing result includes the synchronized data obtained after at least one database completes data synchronization;

[0009] Calculate the field anomaly value of the synchronization data corresponding to each Flink task, and issue an alarm for the Flink task whose field anomaly value of the processing result is greater than the preset threshold.

[0010] In this solution, after receiving the data synchronization instruction and the information of at least one database, at least one flink task is generated. There is a one-to-one correspondence between at least one flink task and at least one database, that is, each flink task is used to perform data synchronization processing on the data of a database, so that the data synchronization processing of each database is independent of each other, which can improve the stability of data synchronization processing; at least one flink task is uniformly submitted to the preset session for parallel processing, and there is no need to set up complex components to process the data of different data sources of each database separately. Through a session channel, parallel processing of data synchronization of all databases can be achieved, thereby improving the efficiency of data synchronization; and, data quality detection, that is, field anomaly detection, is performed on the synchronized data obtained after the data synchronization is completed, and alarm processing is performed on the flink tasks whose field anomaly values ​​are greater than the preset threshold, so as to improve the accuracy of data synchronization.

[0011] Optionally, the information of at least one database includes the identifier of at least one database; in response to the data synchronization instruction, at least one flink task is generated, including: obtaining the configuration information of at least one database according to the identifier of at least one database; for any database of the at least one database, performing the following operations: determining the resource consumption required for the flink task corresponding to any database according to the configuration information of any database, and setting the component memory size and parallelism size of the flink task according to the resource consumption; generating data synchronization rules for any database according to the configuration information and preset requirements of any database.

[0012] Optionally, the preset requirements include at least one of data type requirements, format requirements, name requirements, and association relationship requirements; data synchronization rules for any database are generated based on the configuration information and preset requirements of any database, including: determining at least one operation of extraction, calculation, association, mapping, screening, and conversion to be performed on the data in any database and the order of at least one operation based on at least one of the type, format, name, and association relationship of the data in any database and at least one of the type, format, name, and association relationship requirements of the data set in the preset requirements; and data synchronization rules for any database are generated based on at least one operation and the order of at least one operation.

[0013] Optionally, after submitting at least one flink task to a preset session for parallel processing, the method further includes: obtaining in real time the task status and operating indicators of the preset session when performing data synchronization processing on each flink task; determining whether to adjust at least one of the component memory size and the parallelism size in each flink task according to the operating indicators; and determining whether to interrupt or alarm each flink task according to the task status.

[0014] Optionally, the field outlier value of the synchronization data corresponding to each flink task is calculated, including: for any synchronization data in the synchronization data corresponding to each flink task, performing the following operations: calculating the difference between the local reachability density of the original data before data synchronization corresponding to any synchronization data and the local reachability density of any synchronization data; the local reachability density is used to indicate the density between any data and the neighborhood data of the any synchronization data; if the difference is not 0, calculating the local outlier factor of any synchronization data based on the local density reachability of any synchronization data; the local outlier factor is used to indicate the degree of any synchronization data relative to the domain synchronization data of the any synchronization data; and taking the outlier factor of any synchronization data as the field outlier value of the any synchronization data.

[0015] Optionally, the processing result of at least one flink task also includes the task start time and the task end time of each flink task; after obtaining the processing result of at least one flink task, the method also includes: calculating the time difference between the task start time and the task end time of each flink task, and executing an alarm for the flink task whose time difference is greater than a preset delay.

[0016] Optionally, after obtaining the processing result of at least one flink task, the method also includes: obtaining a first data level of the database corresponding to each flink task, and a second data level of synchronized data obtained after the database completes data synchronization; and executing an alarm for flink tasks whose first data level is different from the second data level.

[0017] In a second aspect, an embodiment of the present application provides a data processing device, including:

[0018] A receiving module, used to: receive a data synchronization instruction and information of at least one database; the data synchronization instruction is used to instruct to perform data synchronization processing on the data of at least one database according to preset requirements;

[0019] A processing module, used for: generating at least one flink task in response to a data synchronization instruction; each flink task in the at least one flink task includes a data synchronization rule corresponding to a preset requirement, and each flink task in the at least one flink task is used to perform data synchronization processing on a corresponding database in the at least one database according to the data synchronization rule; the data synchronization rule in each flink task is different; submitting at least one flink task to a preset session for parallel processing, and obtaining a processing result of the at least one flink task; the processing result includes synchronized data obtained after the at least one database completes data synchronization;

[0020] The detection module is used to calculate the field anomaly value of the synchronization data corresponding to each Flink task, and issue an alarm for the Flink task whose field anomaly value of the processing result is greater than a preset threshold.

[0021] Optionally, the information of at least one database includes the identifier of at least one database; when the processing module generates at least one flink task in response to the data synchronization instruction, it is specifically used to: obtain the configuration information of at least one database according to the identifier of at least one database; for any database among the at least one database, perform the following operations: determine the resource consumption required for the flink task corresponding to any database according to the configuration information of any database, and set the component memory size and parallelism size of the flink task according to the resource consumption; generate data synchronization rules for any database according to the configuration information and preset requirements of any database.

[0022] Optionally, the preset requirements include at least one of data type requirements, format requirements, name requirements, and association relationship requirements; when the processing module generates data synchronization rules for any database based on the configuration information and preset requirements of any database, it is specifically used to: determine at least one operation of extraction, calculation, association, mapping, screening, and conversion to be performed on the data in any database and the order of at least one operation based on at least one of the type, format, name, and association relationship of the data in any database and at least one of the type, format, name, and association relationship requirements of the data set in the preset requirements; generate data synchronization rules for any database based on at least one operation and the order of at least one operation.

[0023] Optionally, after submitting at least one flink task to a preset session for parallel processing, the processing module is further used to: obtain in real time the task status and operating indicators of the preset session when performing data synchronization processing on each flink task; determine whether to adjust at least one of the component memory size and the parallelism size in each flink task according to the operating indicators; and determine whether to interrupt or alarm the execution of each flink task according to the task status.

[0024] Optionally, when calculating the field outlier value of the synchronization data corresponding to each flink task, the detection module is specifically used to: for any synchronization data in the synchronization data corresponding to each flink task, perform the following operations: calculate the difference between the local reachability density of the original data before data synchronization corresponding to any synchronization data and the local reachability density of any synchronization data; the local reachability density is used to indicate the density between any data and the neighborhood data of any synchronization data; if the difference is not 0, calculate the local outlier factor of any synchronization data based on the local density reachability of any synchronization data; the local outlier factor is used to indicate the degree of any synchronization data relative to the domain synchronization data of any synchronization data; and use the outlier factor of any synchronization data as the field outlier value of any synchronization data.

[0025] Optionally, the processing result of at least one flink task also includes the task start time and the task end time of each flink task; after the processing module obtains the processing result of at least one flink task, the detection module is also used to: calculate the time difference between the task start time and the task end time of each flink task, and execute an alarm for the flink task whose time difference is greater than the preset delay.

[0026] Optionally, after the processing module obtains the processing result of at least one flink task, the detection module is also used to: obtain a first data level of the database corresponding to each flink task, and a second data level of the synchronized data obtained after the database completes data synchronization; and execute an alarm for flink tasks whose first data level is different from the second data level.

[0027] In a third aspect, an embodiment of the present application provides an electronic device, comprising at least one processor, wherein the at least one processor is used to execute a computer program stored in a memory so that a method as in the first aspect or any optional implementation of the first aspect is implemented.

[0028] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, which is used to store instructions. When the instructions are executed, the method in the first aspect or any optional implementation of the first aspect is implemented.

[0029] In a fifth aspect, an embodiment of the present application provides a computer program product, including a computer program code, which, when executed on a computer, enables the method in the first aspect or any optional implementation of the first aspect to be implemented.

[0030] The technical effects or advantages of one or more technical solutions provided in the second, third, fourth and fifth aspects of the embodiments of the present application can be correspondingly explained by the technical effects or advantages of one or more corresponding technical solutions provided in the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 A flow chart of a data processing method provided in an embodiment of the present application;

[0032] Figure 2 A flow chart of a method for calculating field outliers of synchronization data provided by an embodiment of the present application;

[0033] Figure 3 An example diagram of an overall framework of data synchronization provided in an embodiment of the present application;

[0034] Figure 4 A structural diagram of a data processing device provided in an embodiment of the present application;

[0035] Figure 5 A structural diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0036] In the technical solution of this application, the collection, dissemination, and use of data are in compliance with the requirements of relevant national laws and regulations.

[0037] It should be noted that in the embodiments of the present application, certain software, components, models and other existing solutions in the industry may be mentioned, and they should be regarded as exemplary. Their purpose is only to illustrate the feasibility of implementing the technical solution of the present application, but it does not mean that the applicant has or will necessarily use the solution.

[0038] The technical solution of the present application is described in detail below through the accompanying drawings and specific embodiments. It should be understood that the embodiments of the present application and the specific features in the embodiments are detailed descriptions of the technical solution of the present application, rather than limitations on the technical solution of the present application. In the absence of conflict, the embodiments of the present application and the technical features in the embodiments can be combined with each other.

[0039] It should be understood that "multiple" in the description of the embodiment of the present application refers to two or more. "First", "second" etc. in the embodiment of the present application are used to distinguish different objects, rather than to describe a specific order. The term "and / or" in the embodiment of the present application is merely a kind of association relationship describing associated objects, indicating that three relationships may exist, for example, A and / or B, which may represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the term "includes" and any of their variations are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device comprising a series of steps or units is not limited to the listed steps or units, but optionally also includes steps or units that are not listed, or optionally also includes other steps or units inherent to these processes, methods, products or devices. The module in the embodiment of the present application refers to a part with independent functions in a software system.

[0040] For ease of understanding, some of the professional terms involved in the embodiments of the present application are explained below:

[0041] 1. Extract, Transform, Load (ETL): It is a technical process widely used in the field of database and data processing. It is mainly used to extract data from different source systems, clean, transform and integrate the data to meet the needs of data analysis and reporting, and finally load the processed data into the target data warehouse or data lake.

[0042] 2. Session environment (Flink deployment mode): In Apache Flink, Session mode is a strategy for deploying and running Flink jobs. In Session mode, you first need to start a Flink cluster and maintain a session. All submitted jobs will run on this cluster. The characteristics of Session mode include: fixed resources, persistent sessions, job independence, and resource release.

[0043] 3. Time Series Database (TSDB): A database designed specifically for storing, querying, and managing time series data. Time series data refers to measurement data recorded in chronological order, usually including a timestamp and one or more numerical data points. The main features and functions of TSDB are to optimize the processing of time series data, including efficient data writing, querying, aggregation, and time window analysis.

[0044] 4. Local Outlier Factor (LOF): It is a density-based anomaly detection algorithm that determines whether each data point is an outlier and the degree of outlier by comparing the density of each data point with its neighborhood points.

[0045] As business complexity increases, the types of data development channel sources in enterprises are also increasing, including MySQL, Redis, Kafka, Hive and other data sources. Efficient synchronization and conversion of multi-source data has become a major challenge. Traditional data synchronization strategies often rely on a series of complex components, such as Canal, Kafka, DataX, Flume, etc. However, this traditional architecture not only increases the complexity and maintenance difficulty of data synchronization, but also makes it difficult to open up data transmission channels when adding new data sources, which complicates the links and significantly increases the development and maintenance costs. It also leads to the dispersion of the technology stack, which is not conducive to unified management and code standardization. In addition, traditional methods lack business agility and intelligent data processing capabilities when facing changes in business needs, making the development process cumbersome and difficult to efficiently process and convert data. In addition, some data transmission methods are limited by the data storage format and cannot even perform simple operations such as data type conversion, which in turn affects the timeliness and accuracy of data-driven decision-making. At the same time, the lack of a comprehensive monitoring and management mechanism makes it difficult to gain insight into quality issues and performance bottlenecks in the data synchronization process. For example, data synchronization tasks are often interrupted or fail due to factors such as cluster resource fluctuations, abnormal data, and changes in source database tables, which seriously affects real-time data services. In addition, it is difficult for operation and maintenance personnel to intuitively grasp the progress and status of synchronization, and they are unable to quickly locate problems and adjust task parameter strategies in a timely manner. As a result, the stability and reliability of data synchronization are insufficient.

[0046] In view of this, an embodiment of the present application is proposed, which generates at least one flink task when a data synchronization instruction and information of at least one database are received, and each flink task is used to perform data synchronization processing on a corresponding database in at least one database, so that the data synchronization of each database is performed independently, thereby improving the stability of data synchronization; submitting at least one flink task to the session for parallel execution does not require reliance on complex components to achieve data synchronization processing for different data sources (i.e., different databases), thereby improving the efficiency of data synchronization; after completing the data synchronization, performing anomaly detection on the synchronized data, calculating its field anomaly value, and executing an alarm for the synchronized data whose field anomaly value is greater than a preset threshold, thereby further ensuring the accuracy of data synchronization.

[0047] The data processing method provided in the embodiment of the present application can be applied to any data synchronization scenario, including but not limited to the above-mentioned data synchronization of business data in different databases of an enterprise.

[0048] It can be understood that before executing the data processing method provided in the embodiment of the present application, it is necessary to deploy the data channel environment first. Specifically, install and deploy flink software and yarn tools on the electronic device used to execute the embodiment of the present application (flink software is used to generate flink tasks according to demand, and yarn tools are used to provide data channels and resource management functions for flink tasks), configure the flink software and the source database and the target database integration related connectors (connectors) into the environment; and start the session on yarn as the data channel, configure the session environment parameters, monitoring functions and automatic restart strategies according to actual needs, and set the sessionID (i.e., the session identifier) ​​in the flink configuration file to ensure that the flink task can be submitted to the corresponding session.

[0049] See also Figure 1 , a data synchronization method provided in an embodiment of the present application, can be applied to any electronic device with processing capability, the method comprising steps S101 to S104:

[0050] S101: Receive a data synchronization instruction and information of at least one database.

[0051] The data synchronization instruction is used to instruct to perform data synchronization processing on the data of at least one database according to preset requirements.

[0052] Exemplarily, the preset requirements include requirements for required synchronization data, which are used to standardize the synchronization data, for example: type requirements, format requirements, name requirements, association relationship requirements, etc. for data such as tables and fields.

[0053] Optionally, the preset requirements may also include requirements for the target database, such as the capacity, type, name, storage location, data storage format, etc. of the target database.

[0054] The information of at least one database may include an identifier of at least one database, such as a number, a storage address, a name, etc. of at least one database, so that relevant configuration information of at least one database can be obtained according to the information of at least one database.

[0055] S102. In response to the data synchronization instruction, generate at least one flink task.

[0056] Among them, each flink task in the at least one flink task includes a data synchronization rule corresponding to the preset requirements, and each flink task in the at least one flink task is used to perform data synchronization processing on a corresponding database in the at least one database according to the data synchronization rule, that is, one flink task corresponds to one database; the data synchronization rule of each flink task is different.

[0057] In a possible embodiment, the information of at least one database includes an identifier of at least one database, and the specific implementation of step S102 is as follows:

[0058] First, according to the identifier of at least one database, configuration information of at least one database is obtained.

[0059] Exemplarily, the configuration information of at least one database includes data storage information of each database in at least one database, such as data type, format, name, association relationship, data magnitude, etc., and application information of the database, such as the connection method of the database (such as the jdbc connection of a MySQL database), the user who creates the database, and the permission password, etc.

[0060] Then, for any database in at least one database, the following operations 1 and 2 are performed, and the order of operation 1 and operation 2 may not be particular:

[0061] Operation 1: Determine the resource consumption required for the Flink task corresponding to any database according to the configuration information of any database, and set the component memory size and parallelism size of the Flink task according to the resource consumption;

[0062] Operation 2: Generate data synchronization rules for any database based on the configuration information and preset requirements of any database.

[0063] Exemplarily, the specific implementation of operation one may be:

[0064] According to the data storage information of any database, such as the data level, preliminarily evaluate the resource consumption required for the flink task corresponding to any database, set the jobmanager and taskmanager memory size of the flink task, and set the parallelism size; among them, the jobmanager is used to split the flink task into multiple subtasks and assign the subtasks to the taskmanager for processing.

[0065] Optionally, you can also set an automatic restart policy for the Flink task. That is, if a task fails or is interrupted, you can re-execute the task according to the automatic restart policy until the number of restarts or failures reaches the restart threshold.

[0066] Exemplarily, the preset requirement includes at least one of a data type requirement, a format requirement, a name requirement, and an association relationship requirement; the specific implementation of the second operation may be:

[0067] Determine, based on at least one of the type, format, name, and association of the data in any database and at least one of the data type requirements, format requirements, name requirements, and association requirements set in the preset requirements, at least one of the operations of extraction, calculation, association, mapping, screening, and conversion to be performed on the data in any database, and the order of the at least one operation;

[0068] A data synchronization rule for any database is generated based on at least one operation and the sequence between at least one operation.

[0069] Optionally, if actual needs require complex logical processing of data in any database, you can also customize a function and add it to the data synchronization rule.

[0070] In this way, the data processing logic (i.e., data synchronization rules) can be customized according to preset requirements and configuration information of each database. Through rich processing functions and conversion engines, complex logical processing can be supported to improve the flexibility and efficiency of data synchronization.

[0071] It can be understood that the operations included in the above data synchronization rules are only a possible example and are not limited to this.

[0072] In a possible embodiment, after the data synchronization rules are generated, the data synchronization rules can also be syntax-checked. If the check fails, the data synchronization rules can be regenerated, or an alert can be sent to a technician, who can modify and confirm the data synchronization rules. In this way, the accuracy and feasibility of the data synchronization rules can be improved, and data synchronization errors or data synchronization failures that cause the restart of the Flink task during subsequent data synchronization processing of the database can be avoided, thereby improving the efficiency and accuracy of data synchronization.

[0073] Optionally, each Flink task also includes configuration information of the target database, such as metadata information of the target database, the name of the target database, etc., so that after the complete data synchronization of each database, the obtained synchronized data can be stored in the target database.

[0074] S103: Submit at least one flink task to a preset session for parallel processing, and obtain the processing result of at least one flink task.

[0075] The processing result includes synchronized data obtained after at least one database completes data synchronization.

[0076] In a possible embodiment, the embodiment of the present application further provides a task monitoring method, that is, after submitting at least one flink task to a preset session for parallel processing, that is, during the process of at least one flink task performing data synchronization processing in the session, the following method is executed:

[0077] Obtain the task status and running indicators of each Flink task when the preset session performs data synchronization processing in real time;

[0078] Determine whether to adjust at least one of the component memory size and parallelism size in each Flink task based on the running indicators;

[0079] According to the task status, determine whether to interrupt or alarm each Flink task.

[0080] Exemplarily, the operating indicators may include: the processing progress of data synchronization, memory, CPU resource usage, etc. of each flink task. If the task processing progress is slow, the memory or CPU resource usage is large, etc., the memory size, parallelism size and other parameters of each component in each flink task can be appropriately adjusted.

[0081] The task status may include: the number of failure alarms, the number of restarts, whether the task is currently in a failure alarm, etc.; if the number of failure alarms or restarts is greater than the preset threshold, or the task is currently in a failure alarm and cannot be automatically restarted, an alarm prompt may be sent to the technician.

[0082] Optionally, you can also monitor the session and flink deployment environment. If there are abnormal parameters in the environment, you can also send an alarm to the technician to ensure the normal progress of the data synchronization task.

[0083] S104: Calculate the field abnormal value of the synchronization data corresponding to each flink task, and issue an alarm for the flink task whose field abnormal value of the synchronization data is greater than a preset threshold.

[0084] Specifically, when calculating the field abnormal value of the synchronization data corresponding to each flink task, the following operations are performed for any synchronization data in the synchronization data corresponding to each flink task:

[0085] Calculate the difference between the local reachability density of the original data before data synchronization corresponding to any synchronized data and the local reachability density of any synchronized data; the local reachability density is used to indicate the density between any data and the neighboring data of the any data;

[0086] If the difference is not 0, then the local outlier factor of any synchronization data is calculated based on the local density reachability of any synchronization data; the local outlier factor is used to indicate the degree of any synchronization data relative to the domain synchronization data of the any synchronization data;

[0087] The outlier factor of any synchronized data is taken as the field outlier of the synchronized data.

[0088] For example, Figure 2 As shown, it is an example flow chart of a method for calculating abnormal field values ​​provided in an embodiment of the present application. The specific implementation of the method is as follows:

[0089] 1. During the execution of each flink task, obtain data monitoring indicators.

[0090] The data monitoring indicators include the synchronization data corresponding to each flink task and the original data before data synchronization corresponding to the synchronization data.

[0091] 2. Store data monitoring indicators in a time series database.

[0092] Taking any flink task in at least one flink task as an example, after the execution of any flink task is completed, the data monitoring indicator corresponding to the any flink task is obtained from the time series database.

[0093] 3. Calculate the k nearest neighbors of the data to be detected (i.e. calculate the k neighborhood data of the data to be detected).

[0094] The data to be detected is the data monitoring index of any flink task, that is, all the synchronization data of any flink task and the original data corresponding to all the synchronization data. The k nearest neighbors of each synchronization data and the k nearest neighbors of the original data corresponding to each synchronization data are calculated respectively to obtain the domain synchronization data of each synchronization data and the domain data of the original data corresponding to each synchronization data.

[0095] 4. Calculate the local reachability density LRD of the data to be detected.

[0096] The LRD of each synchronized data and the LRD of each original data are calculated according to the following formula:

[0097]

[0098] Among them, d(x i ,x j ) represents data x i ,x j The Euclidean distance between k (x j ) represents xj The distance to the kth nearest neighbor of .

[0099] If the difference between the LRD of synchronized data and the LRD of the original data corresponding to the synchronized data is not 0, that is, there is a difference between the density between the synchronized data and its neighboring synchronized data (the density can reflect the distribution of each data), and the density between the original data corresponding to the synchronized data and the neighboring data of the original data, that is, the distribution of the synchronized data is inconsistent with the corresponding original data, and it can be inferred that there may be problems in the synchronization process.

[0100] 5. Calculate the local outlier factor LOF of the data to be detected.

[0101] In the case where the difference between the LRD of the above-mentioned synchronization data and the LRD of the original data corresponding to the synchronization data is not 0, the LOF of the synchronization data is calculated according to the following formula:

[0102]

[0103] The degree of outlier of the synchronization data relative to the neighboring synchronization data can be measured according to the LOF of the synchronization data, and the degree of outlier (ie, LOF) is used as the abnormal field value of the synchronization data.

[0104] 6. Determine whether LOF is less than t. If so, determine that the synchronization data (i.e., the data to be detected) is abnormal data and trigger an alarm. If not, end the detection.

[0105] Where t is the preset threshold. It can be understood that when LOF k (x i ) is significantly greater than 1, it means that the density of the synchronization data in the data distribution is significantly lower than the data distribution density of the adjacent data (i.e., k nearest neighbors), that is, the synchronization data is isolated abnormal data. Therefore, a preset threshold t can be set according to actual needs, and when the synchronization data LOF is greater than the preset threshold, the synchronization data is determined to be abnormal data.

[0106] In this embodiment, by detecting abnormal field values ​​of the synchronization data, abnormal data can be screened out, which is convenient for subsequent technicians to analyze and process the abnormal data, or re-execute the flink task corresponding to the synchronization data to improve the accuracy of data synchronization.

[0107] Optionally, when it is determined that the difference between the LRD of the synchronization data and the LRD of the original data corresponding to the synchronization data is not 0, an alarm may be directly executed on the synchronization data.

[0108] In addition, the embodiments of the present application also provide other methods for detecting whether the synchronization data is abnormal, and the specific methods are shown in Embodiment 1 and Embodiment 2.

[0109] Embodiment 1: The processing result of at least one flink task also includes the task start time and the task end time of each flink task; after obtaining the processing result of at least one flink task, the method further includes:

[0110] Calculate the time difference between the task start time and the task end time of each Flink task, and issue an alarm for Flink tasks whose time difference is greater than the preset delay.

[0111] It can be understood that the preset delay can be set according to user needs and the data level of the database, etc., and the embodiments of the present application do not limit this.

[0112] Embodiment 2: After obtaining the processing result of at least one flink task, the method further includes:

[0113] Obtain the first data level of the database corresponding to each flink task and the second data level of the synchronized data obtained after the database completes data synchronization; and issue an alarm for flink tasks with different first and second data levels.

[0114] It can be understood that the above three embodiments of detecting synchronized data can be used in combination or separately, and can also be combined with other data anomaly detection methods (for example, setting a data identifier for each data in the database, comparing the data identifier of the original data with the data identifier of the corresponding synchronized data, etc.), and the embodiments of the present application do not limit this.

[0115] In this solution, after receiving the data synchronization instruction and the information of at least one database, at least one flink task is generated. There is a one-to-one correspondence between at least one flink task and at least one database, that is, each flink task is used to perform data synchronization processing on the data of a database, so that the data synchronization processing of each database is independent of each other, which can improve the stability of data synchronization processing; at least one flink task is uniformly submitted to the preset session for parallel processing, and there is no need to set up complex components to process the data of different data sources of each database separately. Parallel processing of data synchronization of all databases can be achieved through one session channel, thereby improving the efficiency of data synchronization; and, data quality detection, that is, field anomaly detection, is performed on the synchronized data obtained after the data synchronization is completed, and alarm processing is performed for the flink tasks whose field anomaly values ​​are greater than the preset threshold to ensure the accuracy of data synchronization.

[0116] It is understood that the various embodiments of the data processing method described above can be used alone or in combination. The following provides an example of a general architecture of data synchronization that combines all the above embodiments. Figure 3 , the data synchronization architecture includes:

[0117] Source database, that is, at least one database that needs to be synchronized;

[0118] The target database is the database used to store the synchronization data.

[0119] Development module: includes session environment configuration, data source configuration, data logic processing and task resource parameter configuration.

[0120] Among them, the session environment configuration is used to configure session environment parameters and flink environment parameters, etc.

[0121] Data source configuration is used to obtain the configuration information of the source database and the target database; optionally, the data source configuration can also generate the target database according to the user's requirements;

[0122] Data logic configuration, which is used to generate data synchronization rules according to the user's preset requirements and the configuration information of the source database;

[0123] Task resource parameter configuration is used to set the memory size, parallelism size, etc. of each component of the Flink task according to the configuration information of the source database.

[0124] It is understandable that the development module can also be used to receive instructions.

[0125] The monitoring module includes session environment monitoring, task performance monitoring, task status monitoring, and data quality monitoring.

[0126] Among them, session environment monitoring is used to monitor the environment parameters of the Yarn-session channel and the environment parameters of Flink to ensure the normal progress of data synchronization tasks;

[0127] Task performance monitoring, used to monitor the operating indicators of each Flink task, such as memory, CPU resource usage, task progress, etc.

[0128] Task status monitoring is used to monitor the task status of each Flink task, such as the number of failure alarms, the number of restarts, whether it is currently in normal progress or failure alarm, etc.

[0129] Data quality monitoring is used to detect anomalies in the synchronized data obtained after the Flink task is executed, so as to improve the accuracy of data synchronization.

[0130] The real-time cluster-session mode data channel (Yarn-session channel, referred to as session channel) is used to execute various flink tasks. The upstream data refers to the original data of each database before data synchronization, and the downstream data refers to the synchronized data of each database after data synchronization. The session channel supports parallel processing of multiple flink tasks, which can improve the efficiency of data synchronization.

[0131] It can be understood that in order to achieve data synchronization of at least one database (ie, different data sources in the source database) and improve the efficiency and accuracy of data synchronization, it is possible to use Figure 3 The corresponding method steps in the above corresponding embodiments are executed by each functional module in the embodiment, and the specific method steps in the embodiments of this application will not be repeated here.

[0132] The method provided by the embodiment of the present application is introduced above, and the device provided by the embodiment of the present application is introduced below.

[0133] Based on the same technical concept, an embodiment of the present application provides a data processing device, which includes a module / unit / means for executing the method executed by the electronic device in the above method embodiment. The module / unit / means can be implemented by software, or by hardware, or the corresponding software can be implemented by hardware.

[0134] For example, Figure 4 As shown, the device 400 includes:

[0135] The receiving module 401 is used to: receive a data synchronization instruction and information of at least one database; the data synchronization instruction is used to instruct to perform data synchronization processing on the data of at least one database according to preset requirements;

[0136] The processing module 402 is used to: generate at least one flink task in response to the data synchronization instruction; each flink task in the at least one flink task includes a data synchronization rule corresponding to the preset requirement, and each flink task in the at least one flink task is used to perform data synchronization processing on a corresponding database in the at least one database according to the data synchronization rule; the data synchronization rule in each flink task is different; submit the at least one flink task to the preset session for parallel processing, and obtain the processing result of the at least one flink task; the processing result includes the synchronized data obtained after the at least one database completes the data synchronization;

[0137] The detection module 403 is used to calculate the field abnormality value of the synchronization data corresponding to each flink task, and to issue an alarm to the flink task whose field abnormality value of the processing result is greater than a preset threshold.

[0138] Optionally, the information of at least one database includes the identifier of at least one database; when the processing module 402 generates at least one flink task in response to the data synchronization instruction, it is specifically used to: obtain the configuration information of at least one database according to the identifier of at least one database; for any database in the at least one database, perform the following operations: determine the resource consumption required for the flink task corresponding to any database according to the configuration information of any database, and set the component memory size and parallelism size of the flink task according to the resource consumption; generate data synchronization rules for any database according to the configuration information and preset requirements of any database.

[0139] Optionally, the preset requirements include at least one of data type requirements, format requirements, name requirements, and association relationship requirements; when the processing module 402 generates data synchronization rules for any database based on the configuration information and preset requirements of any database, it is specifically used to: determine at least one operation of extraction, calculation, association, mapping, screening, and conversion to be performed on the data in any database and the order of at least one operation based on at least one of the type, format, name, and association relationship of the data in any database and at least one of the type, format, name, and association relationship requirements of the data set in the preset requirements; generate data synchronization rules for any database based on at least one operation and the order of at least one operation.

[0140] Optionally, after submitting at least one flink task to a preset session for parallel processing, the processing module 402 is further used to: obtain in real time the task status and operating indicators of the preset session when performing data synchronization processing on each flink task; determine whether to adjust at least one of the component memory size and the parallelism size in each flink task according to the operating indicators; and determine whether to interrupt or alarm the execution of each flink task according to the task status.

[0141] Optionally, when calculating the field outlier value of the synchronization data corresponding to each flink task, the detection module 403 is specifically used to: for any synchronization data in the synchronization data corresponding to each flink task, perform the following operations: calculate the difference between the local reachability density of the original data before data synchronization corresponding to any synchronization data and the local reachability density of any synchronization data; the local reachability density is used to indicate the density between any data and the neighborhood data of any synchronization data; if the difference is not 0, calculate the local outlier factor of any synchronization data based on the local density reachability of any synchronization data; the local outlier factor is used to indicate the degree of any synchronization data relative to the domain synchronization data of any synchronization data; and use the outlier factor of any synchronization data as the field outlier value of the any synchronization data.

[0142] Optionally, the processing result of at least one flink task also includes the task start time and the task end time of each flink task; after the processing module 402 obtains the processing result of at least one flink task, the detection module 403 is also used to: calculate the time difference between the task start time and the task end time of each flink task, and execute an alarm for the flink task whose time difference is greater than the preset delay.

[0143] Optionally, after the processing module 402 obtains the processing result of at least one flink task, the detection module 403 is also used to: obtain a first data level of the database corresponding to each flink task, and a second data level of the synchronized data obtained after the database completes data synchronization; and execute an alarm for flink tasks whose first data level is different from the second data level.

[0144] It should be understood that all relevant contents of each step involved in the above method embodiment can be referred to the functional description of the corresponding functional module and will not be repeated here.

[0145] Based on the same technical concept, see Figure 5 The embodiment of the present application further provides an electronic device 500, including:

[0146] At least one processor 501; and a communication interface 503 that is communicatively connected to the at least one processor 501; the at least one processor 501 executes instructions stored in the memory 502, so that the electronic device 500 executes the method steps performed by the billboard in the above method embodiment through the communication interface 503.

[0147] Optionally, the memory 502 is located outside the electronic device 500 .

[0148] Optionally, the electronic device 500 includes the memory 502, the memory 502 is connected to the at least one processor 501, and the memory 502 has instructions that can be executed by the at least one processor 501. Figure 5 The dashed lines indicate that the memory 502 is optional for the electronic device 500 .

[0149] The at least one processor 501 and the memory 502 may be coupled via an interface circuit or may be integrated together, which is not limited here.

[0150] The specific connection medium between the at least one processor 501, the memory 502 and the communication interface 503 is not limited in the embodiment of the present application. Figure 5 In the embodiment, at least one processor 501, a memory 502 and a communication interface 503 are connected via a bus 504. Figure 5The connection between other components is only for schematic illustration and is not intended to be limiting. The bus part may be an address bus, a data bus, a control bus, etc. For ease of representation, Figure 5 It is represented by only one thick line, but it does not mean that there is only one bus or one type of bus.

[0151] It should be understood that the processor mentioned in the embodiments of the present application can be implemented by hardware or by software. When implemented by hardware, the processor can be a logic circuit, an integrated circuit, etc. When implemented by software, the processor can be a general-purpose processor implemented by reading software code stored in a memory.

[0152] Exemplarily, the processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0153] It should be understood that the memory mentioned in the embodiments of the present application may be a volatile memory or a non-volatile memory, or may include both a volatile memory and a non-volatile memory. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which acts as an external cache. By way of example and not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM) and direct memory bus random access memory (DirectRambus RAM, DR RAM).

[0154] It should be noted that when the processor is a general-purpose processor, DSP, ASIC, FPGA or other programmable logic device, discrete gate or transistor logic device, discrete hardware component, the memory (storage module) can be integrated into the processor.

[0155] It should be noted that the memory described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0156] Based on the same technical concept, an embodiment of the present application also provides a computer-readable storage medium, which is used to store instructions. When the instructions are executed, the computer executes the method steps executed by any device in the above method embodiments.

[0157] Based on the same technical concept, an embodiment of the present application also provides a computer program product, including computer program code. When the computer program code is executed on a computer, the method steps executed by any device in the above method embodiment are implemented.

[0158] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0159] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0160] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0161] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0162] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A data processing method, characterized in that: include: Receiving a data synchronization instruction and information of at least one database; the data synchronization instruction is used to instruct to perform data synchronization processing on the data of the at least one database according to preset requirements; In response to the data synchronization instruction, at least one flink task is generated; each of the at least one flink task includes a data synchronization rule corresponding to the preset requirement, and each of the at least one flink task is used to perform data synchronization processing on a corresponding database in the at least one database according to the data synchronization rule; the data synchronization rules in each flink task are different; Submitting the at least one flink task to a preset session for parallel processing, and obtaining a processing result of the at least one flink task; the processing result includes synchronized data obtained after the at least one database completes data synchronization; Calculate the field abnormal value of the synchronization data corresponding to each flink task, and issue an alarm for the flink task whose field abnormal value of the synchronization data is greater than a preset threshold.

2. The method according to claim 1, characterized in that The information of the at least one database includes an identifier of the at least one database; and the generating at least one flink task in response to the data synchronization instruction includes: Acquire configuration information of the at least one database according to the identifier of the at least one database; For any database in the at least one database, perform the following operations: Determine the resource consumption required for the flink task corresponding to any database according to the configuration information of any database, and set the component memory size and parallelism size of the flink task according to the resource consumption; A data synchronization rule for any database is generated according to the configuration information of any database and the preset requirements.

3. The method according to claim 2, characterized in that The preset requirement includes at least one of a data type requirement, a format requirement, a name requirement, and an association requirement; and generating a data synchronization rule for any database according to the configuration information of any database and the preset requirement includes: Determine, based on at least one of the type, format, name, and association of the data in any of the databases and at least one of the data type requirement, format requirement, name requirement, and association requirement set in the preset requirement, at least one of the operations of extraction, calculation, association, mapping, screening, and conversion to be performed on the data in any of the databases, and the order of the at least one operation; A data synchronization rule for any one of the databases is generated based on the at least one operation and the sequence between the at least one operation.

4. The method according to claim 2, characterized in that After submitting the at least one flink task to a preset session for parallel processing, the method includes: Obtain in real time the task status and operation indicators of each flink task when the preset session performs data synchronization processing; Determine whether to adjust at least one of the component memory size and the parallelism size in each flink task according to the operation indicator; According to the task status, determine whether to interrupt or alarm the execution of each flink task.

5. The method according to claim 1, characterized in that The calculating of the field abnormal value of the synchronization data corresponding to each flink task includes: For any synchronized data in the synchronized data corresponding to each flink task, perform the following operations: Calculate the difference between the local reachability density of the original data before data synchronization corresponding to any synchronized data and the local reachability density of any synchronized data; the local reachability density is used to indicate the density between any data and the neighboring data of any data; If the difference is not 0, then calculating the local outlier factor of any synchronization data based on the local density reachability of any synchronization data; the outlier factor is used to indicate the degree of abnormality of any synchronization data relative to the neighboring synchronization data of any synchronization data; The outlier factor of any synchronized data is used as a field outlier value of any synchronized data.

6. The method according to claim 1, characterized in that The processing result of the at least one flink task also includes the task start time and the task end time of each flink task; after obtaining the processing result of the at least one flink task, the method further includes: The time difference between the task start time and the task end time of each flink task is calculated, and an alarm is issued for the flink task whose time difference is greater than a preset delay.

7. The method according to claim 1, characterized in that After obtaining the processing result of the at least one flink task, the method further includes: Obtain a first data level of the database corresponding to each flink task, and a second data level of the synchronized data obtained after the database completes data synchronization; An alarm is issued for a Flink task whose first data level is different from that of the second data level.

8. A data processing device, characterized in that: include: A receiving module, used to: receive a data synchronization instruction and information of at least one database; the data synchronization instruction is used to instruct to perform data synchronization processing on the data of the at least one database according to preset requirements; A processing module, used for: generating at least one flink task in response to the data synchronization instruction; each flink task in the at least one flink task includes a data synchronization rule corresponding to the preset requirement, and each flink task in the at least one flink task is used to perform data synchronization processing on a corresponding database in the at least one database according to the data synchronization rule; the data synchronization rule in each flink task is different; submitting the at least one flink task to a preset session for parallel processing, and obtaining the processing result of the at least one flink task; the processing result includes the synchronization data obtained after the at least one database completes the data synchronization; The detection module is used to calculate the field abnormality value of the synchronization data corresponding to each flink task, and to issue an alarm to the flink task whose field abnormality value of the processing result is greater than a preset threshold.

9. An electronic device, characterized in that: include: A memory for storing program instructions; A processor is used to call the program instructions stored in the memory, and execute the steps included in the method according to any one of claims 1 to 7 according to the obtained program instructions.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium is used to have a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by a computer, the method according to any one of claims 1 to 7 is implemented.

Images