Data synchronization risk identification method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention provides a data synchronization risk identification method and device, electronic equipment and a storage medium, relates to the technical field of data processing, and is suitable for the fields of financial science and technology and medical health. The method comprises the following steps: determining a target table in a target database; determining a predicted incremental data synchronization time range according to the last data synchronization moment of the target table and the data synchronization time interval; obtaining candidate operation logs corresponding to the target table; determining a target operation log in the candidate operation logs within the predicted incremental data synchronization time range according to the target operation; screening the data of the target table according to the target operation log to obtain target change data; and determining non-standard operation data according to at least one of the creation time field and the update time field of the target change data and the predicted incremental data synchronization time range, and performing data synchronization risk alarm. According to the embodiment of the invention, the data synchronization risk identification efficiency and reliability can be improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, and is suitable for the fields of financial technology and medical health, and in particular relates to a data synchronization risk identification method and device, an electronic device and a storage medium. BACKGROUND

[0002] Data synchronization is a technology for maintaining consistency of data among multiple different devices or systems. Specifically, business data in a database of a certain device needs to be synchronized to a data storage system (such as a data lake) of another device. For example, in the bank information storage scenario of the field of financial technology, financial information can be synchronized among different database servers of a distributed server cluster of a bank. For another example, in the medical information storage scenario of the field of medical health, personal medical information of a user can be synchronized across devices.

[0003] Currently, an incremental synchronization method is generally used for data synchronization, that is, only data (such as newly added or modified data) that has changed in the database since the last synchronization is synchronized. This synchronization method requires that operations on data in the database follow a specific operation specification. For example, when adding data in the database, the actual write time of the data needs to be recorded in the database, so as to perform incremental synchronization according to the actual write time. However, there are often data in the database that are not operated according to the specification, such as data that are not recorded with the actual time when added, which causes risks in the data synchronization process. The current method for identifying data that does not conform to the operation specification relies on manual investigation, which is low in efficiency and is easily affected by subjective factors.

[0004] Therefore, how to improve the efficiency and reliability of data synchronization risk identification has become a technical problem to be solved. SUMMARY

[0005] The main purpose of the embodiments of the present application is to propose a data synchronization risk identification method and device, an electronic device and a storage medium, which aims to automatically identify non-standard operation data in a database, so as to improve the reliability and identification efficiency of data synchronization risk identification.

[0006] To achieve the above purpose, a first aspect of the embodiments of the present application proposes a data synchronization risk identification method, which comprises: determining a target table in a target database; wherein the target table has a creation time field and an update time field, and the target table performs incremental data synchronization based on a preset data synchronization time interval; determining a predicted incremental data synchronization time range of the target table according to a time of last data synchronization of the target table and the data synchronization time interval; obtaining an operation log corresponding to the target table to obtain a candidate operation log; filtering the candidate operation logs in the predicted incremental data synchronization time range according to the target operation, to obtain target operation logs; wherein the target operation is used to represent an operation of changing data; filtering data corresponding to the target operation from data of the target table according to the target operation logs, to obtain target changed data; performing time comparison according to at least one of the creation time field and the update time field of the target changed data and the predicted incremental data synchronization time range, to determine non-standard operation data from the target changed data; performing data synchronization risk warning according to the non-standard operation data.

[0007] In some embodiments, the target operation includes an addition operation, and the target changed data corresponding to the addition operation is target addition data; The performing time comparison according to at least one of the creation time field and the update time field of the target changed data and the predicted incremental data synchronization time range, to determine non-standard operation data from the target changed data, includes: performing time range judgment according to the creation time field of the target addition data and the predicted incremental data synchronization time range; if the creation time field of the target addition data does not belong to the predicted incremental data synchronization time range, determining the target addition data as non-standard addition operation data; wherein the non-standard operation data includes the non-standard addition operation data.

[0008] In some embodiments, after the performing time range judgment according to the creation time field of the target addition data and the predicted incremental data synchronization time range, the method further includes: if the creation time field of the target addition data belongs to the predicted incremental data synchronization time range, performing time range judgment according to the update time field of the target addition data and the predicted incremental data synchronization time range; if the update time field of the target addition data does not belong to the predicted incremental data synchronization time range, determining the target addition data as non-standard addition operation data.

[0009] In some embodiments, the target operation includes an update operation, and the target changed data corresponding to the update operation is target update data; The performing time comparison according to at least one of the creation time field and the update time field of the target changed data and the predicted incremental data synchronization time range, to determine non-standard operation data from the target changed data, includes: obtaining a log sending time of the target operation log; performing difference calculation according to the update time field of the target update data and the log sending time, to obtain a target time difference; determining a standard duration threshold according to the duration of the predicted incremental data synchronization time range; wherein the standard duration threshold is less than the duration of the predicted incremental data synchronization time range; if the target time difference is greater than or equal to the standard duration threshold, determining the target update data as non-standard update operation data; wherein the non-standard operation data includes the non-standard update operation data.

[0010] In some embodiments, after the determination of the standard duration threshold according to the duration of the predicted incremental data synchronization time range, the method further comprises: if the target time difference is less than the standard duration threshold, searching for an actual update timestamp of the target update data in the target operation log to obtain a timestamp search result; if the timestamp search result indicates that the actual update timestamp of the target update data exists in the target operation log, and the actual update timestamp is inconsistent with the update time field of the target update data, determining the target update data as non-standard update operation data.

[0011] In some embodiments, the data of the target table has a unique identification field; and the non-standard operation data is at least two. The data synchronization risk warning according to the non-standard operation data comprises: obtaining a total data amount of the non-standard operation data; based on the unique identification field, removing duplicate data from at least two non-standard operation data to obtain deduplicated operation data, and determining the number of the deduplicated operation data as a deduplicated data amount; performing data synchronization risk warning according to the table name of the target table, the total data amount and the deduplicated data amount.

[0012] In some embodiments, after the filtering of the data corresponding to the target operation from the data of the target table according to the target operation log to obtain target change data, the method further comprises: performing time comparison according to the creation time field and the update time field of the target change data; if the creation time field of the target change data is greater than the update time field, determining the target change data as the non-standard operation data.

[0013] To achieve the above object, a second aspect of the embodiment of the present application provides a data synchronization risk identification device, the device comprises: a target table determination module, configured to determine a target table in a target database; wherein the target table has a creation time field and an update time field, and the target table performs incremental data synchronization based on a preset data synchronization time interval; a time range calculation module, configured to determine a predicted incremental data synchronization time range of the target table according to a time of last data synchronization of the target table and the data synchronization time interval; an operation log acquisition module, configured to acquire an operation log corresponding to the target table to obtain a candidate operation log; an operation log screening module, configured to screen the candidate operation log in the predicted incremental data synchronization time range according to a target operation to obtain a target operation log; wherein the target operation is used to represent an operation of changing data; a table data screening module, configured to screen data corresponding to the target operation from data of the target table according to the target operation log to obtain target changed data; a time comparison module, configured to perform time comparison according to at least one of the creation time field and the update time field of the target changed data and the predicted incremental data synchronization time range to determine non-standard operation data from the target changed data; a risk alarm module, configured to perform data synchronization risk alarm according to the non-standard operation data.

[0014] To achieve the above object, a third aspect of the embodiment of the present application provides an electronic device, the electronic device comprises a memory and a processor, the memory stores a computer program, and the processor realizes the method of the first aspect when executing the computer program.

[0015] To achieve the above object, a fourth aspect of the embodiment of the present application provides a computer readable storage medium, the computer readable storage medium stores a computer program, and the computer program is executed by a processor to realize the method of the first aspect.

[0016] The data synchronization risk identification method, device, electronic equipment and storage medium provided by the present application determine a target table in a target database to select a table for subsequent data synchronization. Since the target table performs incremental data synchronization based on a preset data synchronization time interval, i.e., needs to periodically synchronize the changed data in the table, the time range of the predicted incremental data synchronization is determined according to the time when the last data synchronization of the target table was performed and the data synchronization time interval, so that the time range in which the changed data to be synchronized in the target table is determined. The candidate operation logs in the target table are filtered according to the target operation and the candidate operation logs in the predicted incremental data synchronization time range, so that the data corresponding to the target operation in the data of the target table is filtered according to the filtered logs (i.e., the target operation logs), so that the data changed in the target table in the time range (i.e., the predicted incremental data synchronization time range) can be accurately located, such as newly added or modified data, i.e., the data that needs to be analyzed subsequently can be accurately located. At least one of the creation time field and the update time field of the target changed data and the predicted incremental data synchronization time range are compared in time to determine the non-standard operation data from the target changed data. For example, the creation time field and / or the update time field of the target changed data can be compared to determine the non-standard operation data from the target changed data. In this way, the data that does not record the actual creation time or the actual update time field according to the standard can be automatically identified based on the creation time field and / or the update time field, instead of relying on manual subjective investigation. Then, the data synchronization risk alarm is triggered according to the non-standard operation data, so that the data synchronization risk caused by non-standard operation can be efficiently identified, such as the risk of missing incremental synchronization caused by not recording the actual creation (or update) time, thereby improving the automation degree of risk identification and improving the efficiency and reliability of risk identification. BRIEF DESCRIPTION OF DRAWINGS

[0017] Figure 1 is a flowchart of the data synchronization risk identification method provided by an embodiment of the present application; Figure 2 is a flowchart of step 106 in Figure 1 ; Figure 3 is a flowchart of the data synchronization risk identification method provided by another embodiment of the present application; Figure 4 is a flowchart of step 302 in Figure 3 ; Figure 5 is a flowchart of the data synchronization risk identification method provided by another embodiment of the present application; Figure 6 is a flowchart of step 107 in Figure 1 ; Figure 7 is a flowchart of a data synchronization risk identification method provided by another embodiment of the present application; Figure 8 is a structural schematic diagram of a data synchronization risk identification device provided by an embodiment of the present application; Figure 9 is a hardware structural schematic diagram of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0018] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in combination with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application.

[0019] It should be noted that although the functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in a manner different from the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the specification and claims and the above-described drawings are used to distinguish similar objects, and do not necessarily describe a specific order or sequence.

[0020] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present application belongs. The terms used herein are only for the purpose of describing the embodiments of the present application and are not intended to limit the present application.

[0021] First, the meanings of several terms involved in the present application are analyzed: Business database: is a data storage system specially used for storing, managing and processing enterprise daily operation data. The business database is mainly used for storing structured data related to business activities, such as customer information, order records, inventory data, financial information, etc.

[0022] Data lake (Data Lake): is a data storage system used for storing and managing diversified data. Data lake can be used to store data in raw format. For example, data lake can store both structured data (such as table data in relational database) and unstructured raw data (such as documents in different formats).

[0023] Data synchronization: is a technology that keeps the consistency of data state among multiple systems through technical means. The main purpose of data synchronization technology is to keep the consistency of data among multiple different devices or systems. The way of data synchronization includes full data synchronization and incremental data synchronization.

[0024] Full data synchronization: is to synchronize all data in the source data storage system to the target data storage system according to a synchronization period. Full data synchronization is the simplest way to ensure data synchronization on both sides, but the amount of synchronized data is large, which consumes a large amount of CPU resources and memory resources.

[0025] Incremental data synchronization: refers to synchronizing only the changed data in the source data storage system to the target data storage system in each synchronization period. For example, only synchronize the data (such as newly added or modified data) that has changed in the database since the last synchronization.

[0026] The business database of an enterprise stores business data, and the data lake can periodically (such as every day) read snapshot data of the business database for data synchronization. If the incremental synchronization method is used to synchronize the data in the business database to the data lake, the operation specification requirement for the data change operation of the business database needs to be proposed, otherwise it is impossible to guarantee that the data lake synchronously obtains all the latest data in the business database, which will cause data synchronization loss. The operation specification is as follows: 1. The creation time field in the table of the business database needs to take the system time (i.e. the actual time) when the data row falls into the database, and cannot be artificially specified as other time; 2. The update time field in the table of the business database needs to take the system time when the data is modified, and cannot be artificially specified as other time, nor can the update time field be updated when the data is modified.

[0027] However, at present, the data change operation of the business database often does not follow the above operation specification. For example, new data is added in the business database, but the creation time is not taken as the system time when the data is stored in the database, or the data is updated in the business database, but the system time when the update operation is performed is not written into the update time field. This will cause the data lake to be unable to synchronize all the latest data, resulting in risks in the data synchronization process. It is currently difficult to identify such data synchronization risks. The current method relies on manual investigation to identify data that does not comply with the operation specification, in order to identify data synchronization risks, but this method is low in efficiency and is easily affected by subjective factors. Especially for businesses with high data synchronization quality requirements, such as regulatory reporting and financial-related businesses, the data between the business database and the data lake needs to be kept highly consistent, and the current method is difficult to meet the above quality requirements.

[0028] Based on this, the embodiments of the present application propose a data synchronization risk identification method and device, an electronic device and a storage medium, aiming to improve the efficiency and reliability of data synchronization risk identification. In view of the defect that the non-standard operation of the business database causes the data lake to fail to synchronize the latest version of the data, the embodiments of the present application can automatically calculate the number of non-standard operation behaviors of the business database and the impact range, uniformly identify the data synchronization risk of the business database, and expose the possible risks of the synchronized data in advance.

[0029] The data synchronization risk identification method, device, electronic device, and storage medium provided by the embodiments of the present application are described in detail as follows. First, the data synchronization risk identification method in the embodiments of the present application is described.

[0030] The data synchronization risk identification method provided by the embodiments of the present application can be applied to a terminal, can be applied to a server end, and can also be software running in a terminal or a server end. In some embodiments, the terminal can be a smart phone, a tablet computer, a notebook computer, a desktop computer, or the like; the server end can be configured as a stand-alone physical server, can be configured as a server cluster or a distributed system formed by multiple physical servers, can also be configured as a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms; and the software can be an application that implements the data synchronization risk identification method, but is not limited to the above forms.

[0031] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld devices or portable devices, tablet devices, multi-processor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, and the like. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, and the like that perform specific tasks or implement specific abstract data types. The present application can also be practiced in a distributed computing environment, in which tasks are performed by remote processing devices connected by a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media, including storage devices.

[0032] It should be noted that in each specific embodiment of the present application, when relevant processing needs to be performed according to user information, user behavior data, user historical data, and user location information, and the like related to the identity or characteristics of the user, the user's permission or consent will be obtained first, and the collection, use, and processing of the data will comply with relevant laws, regulations, and standards. In addition, when the embodiments of the present application need to obtain sensitive personal information of the user, the separate permission or separate consent of the user will be obtained through a pop-up window or a jump to a confirmation page, and after obtaining the separate permission or separate consent of the user, the necessary user-related data for enabling the embodiments of the present application to operate normally will be obtained.

[0033] Figure 1 is an optional flowchart of a data synchronization risk identification method provided by an embodiment of the present application, Figure 1 The method in the method can include but is not limited to steps 101-107.

[0034] Step 101, determine a target table in a target database; wherein the target table has a creation time field and an update time field, and the target table performs incremental data synchronization based on a preset data synchronization time interval; Step 102, determine a predicted incremental data synchronization time range of the target table according to a time of last data synchronization of the target table and the data synchronization time interval; Step 103, obtain an operation log corresponding to the target table to obtain a candidate operation log; Step 104, filter the candidate operation log in the predicted incremental data synchronization time range according to a target operation to obtain a target operation log; wherein the target operation is used to represent an operation of changing data; Step 105, filter data corresponding to the target operation from the data of the target table according to the target operation log to obtain target changed data; Step 106, perform time comparison according to at least one of the creation time field and the update time field of the target changed data and the predicted incremental data synchronization time range to determine non-standard operation data from the target changed data; Step 107, perform data synchronization risk warning according to the non-standard operation data.

[0035] The beneficial effects of the embodiments of the present application include but are not limited to: determining a target table in a target database to select a table for subsequent data synchronization. Since the target table performs incremental data synchronization based on a preset data synchronization time interval, that is, the changed data in the table needs to be synchronized periodically, the time range of the predicted incremental data synchronization is determined according to the time when the last data synchronization of the target table is performed and the data synchronization time interval, so that the time range in which the changed data to be synchronized in the target table is determined. By filtering the candidate operation logs in the predicted incremental data synchronization time range from the candidate operation logs corresponding to the target table according to the target operation, the data corresponding to the target operation in the data of the target table is filtered according to the filtered logs (i.e., target operation logs), so that the data changed in the target table in the time range (i.e., the predicted incremental data synchronization time range) can be accurately located, such as newly added or modified data, that is, the data that needs to be analyzed subsequently can be accurately located. At least one of the creation time field and the update time field of the target changed data and the predicted incremental data synchronization time range are compared in time to determine the non-standard operation data from the target changed data. For example, the creation time field and / or the update time field of the target changed data can be compared to determine whether it belongs to the predicted incremental data synchronization time range, so as to determine the non-standard operation data from the target changed data. In this way, the data that does not record the actual creation time or update time field according to the standard can be automatically identified based on the creation time field and / or the update time field, rather than relying on manual subjective investigation. Then, the data synchronization risk alarm is triggered according to the non-standard operation data, so that the data synchronization risk caused by non-standard operation can be efficiently identified, such as the risk of missing incremental synchronization caused by not recording the actual creation (or update) time, thereby improving the automation degree of risk identification to improve the efficiency and reliability of risk identification.

[0036] In step 101 of some embodiments, the target table refers to a table in the target database that needs to perform incremental data synchronization. The number of target tables can be one or more, which is not limited by the embodiments of the present application. It should be noted that the data synchronization time interval is the synchronization period of the target table, for example, the data synchronization time interval can include any of the following: one day, one week, one month, etc. In addition, the time unit of the data synchronization time interval can also be other time units, such as hours or minutes, without limitation.

[0037] In some embodiments, specifically, the target database can be a business database. For example, in a bank information storage scenario in the field of financial technology, the target database can be a financial business database of a bank, and the target table in the financial business database can be used to store personal financial information of a user. For another example, in a medical information storage scenario in the field of medical health, the target database can be a medical business database of a bank, and the target table in the medical business database can be used to store medical data of a hospital.

[0038] In step 102 of some embodiments, the predicted incremental data synchronization time range is a time range formed from a time point of a last data synchronization of the target table (i.e., a time point of a latest data synchronization) to a time point after the data synchronization time interval. For example, assuming that the data synchronization time interval is 1 hour, and the time point of the last data synchronization of the target table is 14:00:00 (hour:minute:second) on date D1, then the predicted incremental data synchronization time range is from 14:00:00 on date D1 to 15:00:00 on the same day.

[0039] In step 103 of some embodiments, the candidate operation log is a database operation log corresponding to the target table. It should be noted that the database operation log is a log file used to record all data operation activities (such as data creation, update, deletion, etc.) in a database system. Specifically, the candidate operation log records data operation activities related to the target table.

[0040] In step 104 of some embodiments, the target operation is an operation of changing data in a table of a database, such as an addition operation, an update operation, etc. The target operation log is information of the candidate operation log recording the target operation in the predicted incremental data synchronization time range. Specifically, assuming that the data synchronization period is every day, then the database operation log can be read by a timing scheduling system every day to divide log data corresponding to the addition operation or the update operation.

[0041] In some embodiments, for example, assuming that the predicted incremental data synchronization time range is from 14:00:00 to 15:00:00 on date D1, and the candidate operation log of the target table includes log Log1 and log Log2. The log Log1 records log information between 14:00:00 on date D1 and 15:00:00 on the same day, and the log Log2 records log information on date D2 (date D2 is different from date D1), then the log Log1 is determined as the target operation log.

[0042] In step 105 of some embodiments, the target changed data is data in the target table that is changed (e.g., added or updated) within the expected incremental data synchronization time range, that is, data corresponding to an add operation or an update operation performed on the target table within the expected incremental data synchronization time range. Specifically, the target operation can include an add operation and an update operation.

[0043] In step 106 of some embodiments, the non-standard operation data refers to data corresponding to a non-standard target operation in the target table. For a specific process of determining the non-standard operation data through time ratio comparison, refer to the specific description of steps 201 to 202 and steps 401 to 404 below, which will not be repeated here.

[0044] In step 107 of some embodiments, an alarm email can be generated according to the non-standard operation data, and the alarm email can be sent to the mailbox of the operation and maintenance personnel of the target database, so that manual intervention can be prompted when the data synchronization risk is identified, so as to perform subsequent data difference investigation and data repair.

[0045] In some embodiments, the number of target tables is multiple, and the naming of the creation time field and / or the update time field between the multiple target tables can be different. For example, assuming that the multiple target tables include table A, table B and table C, the naming of the creation time field of table A is created_date, the naming of the creation time field of table B is date_created, and the naming of the creation time field of table C is created_time, it can be seen that the naming of the creation time field of the three is not the same. In order to uniformly obtain the fields of different target tables, the mapping relationship between the table name and the field naming of each target table can be stored in advance, such as storing the mapping relationship in a list or a table, so as to facilitate subsequent obtaining of the creation time field and / or the update time field of the data in the multiple target tables.

[0046] Please refer to Figure 2 In some embodiments, the target operation includes an add operation, and the target changed data corresponding to the add operation is target added data. Step 106 can include but is not limited to steps 201 to 202: Step 201, time range judgment is performed according to the creation time field of the target added data and the expected incremental data synchronization time range. Step 202, if the creation time field of the target added data does not belong to the expected incremental data synchronization time range, the target added data is determined as non-standard add operation data; wherein the non-standard operation data includes non-standard add operation data.

[0047] The embodiment has the advantages that, after the actual new operation recorded in the operation log within the synchronization time range (i.e., the expected incremental data synchronization time range) is operated, the data newly added to the target table within the synchronization time range is determined, and then the time range judgment is performed according to the creation time field of the target new data and the expected incremental data synchronization time range. In this way, it can be determined whether the creation time (i.e., the value of the creation time field) of the data stored in the target table belongs to the synchronization time range. If the creation time field of the target new data does not belong to the expected incremental data synchronization time range, it indicates that the new time of the data stored in the table is not the actual new time. Therefore, the target new data is determined as the non-standard new operation data. In this way, the data synchronization risk caused by the non-standard new operation of the database can be efficiently identified, that is, the risk of missing incremental synchronization caused by the fact that the data does not record the actual creation time can be identified. Therefore, the automation degree of risk identification is improved, and the risk identification efficiency and reliability are improved.

[0048] In step 201 of some embodiments, it can be determined whether the creation time field of the target new data belongs to the expected incremental data synchronization time range.

[0049] In step 202 of some embodiments, the non-standard operation data includes non-standard new operation data. Specifically, the non-standard new operation data is the non-standard operation data corresponding to the new operation.

[0050] Please refer to Figure 3 In some embodiments, after step 201, the data synchronization risk identification method can further include but is not limited to steps 301 to 302: Step 301, if the creation time field of the target new data belongs to the expected incremental data synchronization time range, performing time range judgment according to the update time field of the target new data and the expected incremental data synchronization time range; Step 302, if the update time field of the target new data does not belong to the expected incremental data synchronization time range, determining the target new data as non-standard new operation data.

[0051] The embodiment has the advantages that, after it is detected that the creation time field of the target new data belongs to the expected incremental data synchronization time range, it is further detected whether the update time field of the target new data belongs to the expected incremental data synchronization time range, so as to further improve the comprehensiveness of data synchronization risk identification. In this way, the target new data can be determined as non-standard new operation data when either the creation time field or the update time field of the data does not belong to the synchronization time range. In this way, the data synchronization risk caused by the non-standard new operation of the database can be efficiently identified, that is, the risk of missing incremental synchronization caused by the fact that the data does not record the actual creation time can be identified. Therefore, the automation degree of risk identification is improved, and the identification efficiency is improved.

[0052] In step 301 of some embodiments, it can be judged whether the creation time field and the update time field of the target new data both belong to the expected incremental data synchronization time range.

[0053] In step 302 of some embodiments, if any one of the creation time field and the update time field of the target new data does not belong to the expected incremental data synchronization time range, the target new data is determined as non-standard new operation data.

[0054] Please refer to Figure 4 In some embodiments, the target operation includes an update operation, and the target change data corresponding to the update operation is target update data. Step 302 can include but is not limited to steps 401 to 404: Step 401, obtaining the log sending time of the target operation log; Step 402, calculating the target time difference according to the difference between the update time field of the target update data and the log sending time; Step 403, determining a standard duration threshold according to the duration of the expected incremental data synchronization time range; wherein the standard duration threshold is less than the duration of the expected incremental data synchronization time range; Step 404, if the target time difference is greater than or equal to the standard duration threshold, determining the target update data as non-standard update operation data; wherein the non-standard operation data includes the non-standard update operation data.

[0055] The advantage of this embodiment is that after determining the data (i.e. target update data) updated in the target table within the synchronization time range (i.e. expected incremental data synchronization time range) recorded in the operation log within the synchronization time range, the log sending time of the target operation log is obtained, and the target time difference is calculated according to the update time field of the target update data and the log sending time. Considering that the target database may have transactions with long execution time, the target database may generate and send logs after a period of time after data update, but the time difference between the log sending time and the data update time should not be too large, such as at least not more than the duration of the expected incremental data synchronization time range (hereinafter referred to as synchronization time range). Therefore, a standard duration threshold is determined based on the duration of the synchronization time range, and in the case where the target time difference is greater than or equal to the standard duration threshold, the target update data is determined as non-standard update operation data. This can efficiently identify the data synchronization risk caused by non-standard update operation of the database, i.e. identify the risk of incremental synchronization omission caused by not recording the actual update time, thereby improving the automation degree and efficiency of risk identification.

[0056] In step 401 of some embodiments, the target database can automatically generate the operation log by a log generation plug-in or the like, and send the operation log to other synchronization devices.

[0057] In step 402 of some embodiments, the target time difference is an absolute value of a time difference between an update time field of the target update data and the log sending time.

[0058] In step 403 of some embodiments, the standard duration threshold is less than a duration of the predicted incremental data synchronization time range. For example, assuming that the duration of the predicted incremental data synchronization time range is 1 day, the standard duration threshold can be 4 hours. The standard duration threshold can also be other numerical values, not limited thereto.

[0059] In step 404 of some embodiments, the non-standard operation data includes non-standard update operation data. Specifically, the non-standard update operation data is the non-standard operation data corresponding to the update operation.

[0060] Please refer to Figure 5 In some embodiments, after step 403, the data synchronization risk identification method can further include but is not limited to steps 501-502: Step 501, if the target time difference is less than the standard duration threshold, searching for an actual update timestamp of the target update data in the target operation log to obtain a timestamp search result. Step 502, if the timestamp search result indicates that the actual update timestamp of the target update data exists in the target operation log and is inconsistent with the update time field of the target update data, determining the target update data as non-standard update operation data.

[0061] The advantage of this embodiment is that by searching for the actual update timestamp of the target update data in the target operation log when the target time difference is less than the standard duration threshold, the non-standard update operation data is more accurately identified in the case that the actual update timestamp exists in the target operation log. If the timestamp search result indicates that the actual update timestamp of the target update data exists in the target operation log and is inconsistent with the update time field of the target update data, the target update data is determined as non-standard update operation data. In this way, the data synchronization risk caused by the non-standard update operation of the database can be more accurately identified, thereby improving the reliability of risk identification.

[0062] In step 501 of some embodiments, considering that the log can include the actual update timestamp of the data in the database, the timestamp search is performed on the target operation log. Specifically, the timestamp search result can be that the actual update timestamp of the target update data exists in the target operation log, or that the actual update timestamp of the target update data does not exist in the target operation log.

[0063] In step 502 of some embodiments, if the actual update timestamp is inconsistent with the update time field of the target update data, the actual update timestamp of the target update data can be written into the update time field of the target update data.

[0064] Please refer to Figure 6 In some embodiments, the data of the target table has a unique identification field; and the non-standard operation data is at least two; Step 107 can include but is not limited to steps 601-603: Step 601, obtaining the total data amount of the non-standard operation data; Step 602, based on the unique identification field, removing duplicate data from the at least two non-standard operation data to obtain deduplicated operation data, and determining the number of the deduplicated operation data as the deduplicated data amount; Step 603, performing data synchronization risk warning according to the table name of the target table, the total data amount and the deduplicated data amount.

[0065] The advantage of this embodiment is that by obtaining the total data amount of the non-standard operation data, specific information for risk warning is obtained. Based on the unique identification field, duplicate data is removed from the at least two non-standard operation data to obtain deduplicated operation data, and the number of the deduplicated operation data is determined as the deduplicated data amount, so that the data amount of the target table actually changed (such as added or updated) within the expected incremental data synchronization time range can be determined. Then, according to the table name of the target table, the total data amount and the deduplicated data amount, data synchronization risk warning is performed, so that automatic early warning can be performed for the identified data synchronization risk, and the comprehensiveness of risk warning is improved.

[0066] In step 601 of some embodiments, it should be noted that at the same time, the unique identification fields of any two data in the same target table are different. Specifically, the unique identification field can be the primary key of the target table.

[0067] In step 602 of some embodiments, it is to be noted that data with the same unique identifier field in the target table at different time points can not be completely consistent data. For example, assume that a data DATA1 with a primary key J1 is first added to the target table, then the data DATA1 is deleted, and then a data DATA2 with the primary key J1 is added again. Assume that the data DATA1 and the data DATA2 are both non-standard operation data, then although the primary keys of the data DATA1 and the data DATA2 are the same, other fields can be different, that is, the above two pieces of data can not be completely consistent. Moreover, in the embodiments of the present application, two addition operations are performed on the target table, but finally only one piece of non-standard data is added to the target table. In this case, the deduplicated data amount is used to represent the number of non-standard changed data finally possessed by the target table after one or more change operations on the target table within the predicted incremental data synchronization time range.

[0068] In step 603 of some embodiments, the table name, the total data amount and the deduplicated data amount of the target table can be stored in a document or a statistical table. An alarm email can also be generated according to the table name, the total data amount and the deduplicated data amount of the target table.

[0069] Please refer to Figure 7 In some embodiments, after step 105, the data synchronization risk identification method can further include but is not limited to steps 701 to 702: Step 701, performing time comparison according to the creation time field and the update time field of the target changed data; Step 702, if the creation time field of the same target changed data is greater than the update time field, determining the target changed data as non-standard operation data.

[0070] The advantage of this embodiment is that the creation time field and the update time field of the target changed data are compared, and if the creation time field of the same target changed data is greater than the update time field, the target changed data is determined as non-standard operation data. In this way, whether the creation time of the same data is earlier than the update time can be used to identify non-standard operation data, thereby improving the comprehensiveness of data synchronization risk identification.

[0071] In step 701 of some embodiments, for the same target changed data, whether the value of the creation time field is less than or equal to the value of the update time field, that is, whether the creation time of the target changed data is earlier than or equal to the update time, can be compared.

[0072] In step 702 of some embodiments, if the creation time field of the same target changed data is greater than the update time field, the target changed data does not conform to the rules of the time sequence logic, and the target changed data can be determined as non-standard operation data.

[0073] Referring to Figure 8 The embodiment of the application further provides a data synchronization risk identification device, which can implement the above-mentioned data synchronization risk identification method. The device comprises: A target table determination module 801 is configured to determine a target table in a target database; wherein the target table has a creation time field and an update time field, and the target table performs incremental data synchronization based on a preset data synchronization time interval; A time range calculation module 802 is configured to determine a predicted incremental data synchronization time range of the target table according to a time of last data synchronization of the target table and the data synchronization time interval; An operation log acquisition module 803 is configured to acquire an operation log corresponding to the target table to obtain a candidate operation log; An operation log screening module 804 is configured to screen the candidate operation log in the predicted incremental data synchronization time range according to a target operation to obtain a target operation log; wherein the target operation is used to represent an operation of changing data; A table data screening module 805 is configured to screen data corresponding to the target operation from data of the target table according to the target operation log to obtain target changed data; A time comparison module 806 is configured to compare time according to at least one of the creation time field and the update time field of the target changed data and the predicted incremental data synchronization time range to determine non-standard operation data from the target changed data; A risk warning module 807 is configured to perform data synchronization risk warning according to the non-standard operation data.

[0074] In an embodiment, the data synchronization risk identification device further comprises a first non-standard data screening module, which is configured to: if the creation time field of the target new data belongs to the predicted incremental data synchronization time range, perform time range judgment according to the update time field of the target new data and the predicted incremental data synchronization time range; and if the update time field of the target new data does not belong to the predicted incremental data synchronization time range, determine the target new data as non-standard new operation data.

[0075] In an embodiment, the data synchronization risk identification device further comprises a second non-standard data screening module, which is configured to: if the target time difference is less than the standard duration threshold, find an actual update timestamp of the target update data in the target operation log to obtain a timestamp finding result; and if the timestamp finding result indicates that the actual update timestamp of the target update data exists in the target operation log and is inconsistent with the update time field of the target update data, determine the target update data as non-standard update operation data.

[0076] In an embodiment, the data synchronization risk identification apparatus further comprises a third non-standard data screening module configured to: perform a time comparison according to the creation time field and the update time field of the target change data; and determine the target change data as non-standard operation data if the creation time field of the same target change data is greater than the update time field.

[0077] The specific embodiments of the data synchronization risk identification apparatus are basically the same as the specific embodiments of the data synchronization risk identification method described above, and thus will not be described again here.

[0078] The embodiments of the present application further provide an electronic device, which comprises a memory and a processor. The memory stores a computer program, and the processor implements the data synchronization risk identification method described above when executing the computer program. The electronic device can include any smart terminal such as a tablet computer or a vehicle-mounted computer.

[0079] Please refer to Figure 9 , Figure 9 The hardware structure of the electronic device of another embodiment is illustrated, which comprises: The processor 901 can be implemented in a general-purpose CPU (Central Processing Unit), a microprocessor, an ASIC (Application Specific Integrated Circuit), or one or more integrated circuits, and is used to execute related programs to implement the technical solutions provided by the embodiments of the present application. The memory 902 can be implemented in the form of a ROM (Read-Only Memory), a static storage device, a dynamic storage device, or a RAM (Random Access Memory). The memory 902 can store an operating system and other application programs. When the technical solutions provided by the embodiments of the present application are implemented by software or firmware, the related program codes are stored in the memory 902 and are called and executed by the processor 901 to implement the data synchronization risk identification method of the embodiments of the present application. The input / output interface 903 is used to realize information input and output. The communication interface 904 is used to realize the communication interaction between the device and other devices. The communication can be realized by a wired manner (such as USB, network cable, etc.) or a wireless manner (such as mobile network, WIFI, Bluetooth, etc.). The bus 905 transmits information between various components (such as the processor 901, the memory 902, the input / output interface 903, and the communication interface 904) of the device. The processor 901, the memory 902, the input / output interface 903, and the communication interface 904 are communicatively connected with each other through the bus 905.

[0080] The application also provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the data synchronization risk identification method.

[0081] The memory is a non-transitory computer readable storage medium, and can be used to store non-transitory software programs and non-transitory computer executable programs. In addition, the memory can include a high-speed random access memory, and can also include a non-transitory memory, such as at least one magnetic disk storage device, a flash memory device, or other non-transitory solid-state memory device. In some embodiments, the memory can optionally include a memory remotely arranged relative to the processor, and these remote memories can be connected to the processor through a network. Examples of the above network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0082] It should be noted that the non-company software tools or components appearing in the embodiments of the present application are only examples for introduction, and do not represent actual use.

[0083] The embodiments described in the present application are used to more clearly illustrate the technical solutions of the embodiments of the present application, and do not constitute a limitation on the technical solutions provided by the embodiments of the present application. Those skilled in the art can know that, with the evolution of technology and the appearance of new application scenarios, the technical solutions provided by the embodiments of the present application are also applicable to similar technical problems.

[0084] Those skilled in the art can understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and can include more or fewer steps than the figures shown, or combine certain steps, or different steps.

[0085] The device embodiments described above are only schematic, and the units described as separate components can or can not be physically separate, that is, can be located in one place, or can be distributed on multiple network units. Part or all of the modules can be selected according to actual needs to achieve the purpose of the embodiments of the present application.

[0086] Those skilled in the art can understand that all or some steps in the above disclosed method, the function modules / units in the system and the device can be implemented as software, firmware, hardware and their appropriate combinations.

[0087] The terms "first", "second", "third", "fourth", and the like in the description of this application and in the claims hereof, if any, are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It is to be understood that the use of the terms so termed herein is solely for the convenience of the reader and does not limit the scope of the application. It is also to be understood that the description and examples in this application are intended to cover all possible combinations where any of the several elements can represent one or more elements.

[0088] It should be understood that, in the application, "at least one" refers to one or more, and "multiple" refers to two or more. "And / or" is used to describe the relationship between associated objects, which means that there can be three relationships, for example, "A and / or B" can represent three cases: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally represents an "or" relationship between the associated objects. "At least one of the following" or similar expressions means any combination of these items, including any combination of single or multiple items. For example, at least one of a, b or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0089] In several embodiments provided in the application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only illustrative, for example, the division of the above-mentioned units is only a logical functional division, and actual implementation can have another division manner, for example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. The coupling or direct coupling or communication connection between the displayed or discussed each other can be indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0090] The units described above as separate components can or can not be physically separated, and the components shown as units can or can not be physical units, that is, they can be located in one place, or they can be distributed on multiple network units. According to actual needs, part or all of the units can be selected to achieve the purpose of the embodiment of the application.

[0091] In addition, each function unit in each embodiment of the present application can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software function unit.

[0092] When the integrated unit is realized in the form of a software function unit and sold or used as an independent product, it can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application, essentially or in part, or all or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, and includes multiple instructions used to cause a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods in the embodiments of the present application. The foregoing storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various program storage media.

[0093] The preferred embodiments of the embodiments of the present application are described above with reference to the accompanying drawings, and are not limited to the scope of the embodiments of the present application. Any modifications, equivalent replacements and improvements made by those skilled in the art without departing from the scope and essence of the embodiments of the present application shall be within the scope of the embodiments of the present application.

Claims

1. A method for identifying data synchronization risks, characterized in that, The method comprises: determining a target table in a target database; wherein the target table has a creation time field and an update time field, and the target table performs incremental data synchronization based on a preset data synchronization time interval; determining a predicted incremental data synchronization time range of the target table according to a time of last data synchronization of the target table and the data synchronization time interval; obtaining operation logs corresponding to the target table to obtain candidate operation logs; screening the candidate operation logs in the predicted incremental data synchronization time range according to a target operation to obtain target operation logs; wherein the target operation represents an operation of changing data; screening data corresponding to the target operation from data of the target table according to the target operation logs to obtain target change data; performing time comparison on at least one of the creation time field and the update time field of the target change data and the predicted incremental data synchronization time range to determine non-standard operation data from the target change data; performing data synchronization risk warning according to the non-standard operation data.

2. The method of claim 1, wherein, The target operation comprises an addition operation, and the target change data corresponding to the addition operation is target addition data; The method of performing time comparison on at least one of the creation time field and the update time field of the target change data and the predicted incremental data synchronization time range to determine non-standard operation data from the target change data comprises: performing time range judgment on the creation time field of the target addition data and the predicted incremental data synchronization time range; if the creation time field of the target addition data does not belong to the predicted incremental data synchronization time range, determining the target addition data as non-standard addition operation data; wherein the non-standard operation data comprises the non-standard addition operation data.

3. The method of claim 2, wherein, After the time range judgment on the creation time field of the target addition data and the predicted incremental data synchronization time range, the method further comprises: if the creation time field of the target addition data belongs to the predicted incremental data synchronization time range, performing time range judgment on the update time field of the target addition data and the predicted incremental data synchronization time range; if the update time field of the target addition data does not belong to the predicted incremental data synchronization time range, determining the target addition data as non-standard addition operation data.

4. The method of claim 1, wherein, The target operation comprises an update operation, and the target change data corresponding to the update operation is target update data; The method of performing time comparison on at least one of the creation time field and the update time field of the target change data and the predicted incremental data synchronization time range to determine non-standard operation data from the target change data comprises: obtaining a log sending time of the target operation logs; performing difference calculation on the update time field of the target update data and the log sending time to obtain a target time difference; determine a standard duration threshold according to the duration of the predicted incremental data synchronization time range, wherein the standard duration threshold is less than the duration of the predicted incremental data synchronization time range; if the target time difference is greater than or equal to the standard duration threshold, determine the target update data as non-standard update operation data; wherein the non-standard operation data includes the non-standard update operation data.

5. The method of claim 4, wherein, After the method of determining a standard duration threshold according to the duration of the predicted incremental data synchronization time range, the method further comprises: if the target time difference is less than the standard duration threshold, find the actual update timestamp of the target update data in the target operation log to obtain a timestamp search result; if the timestamp search result indicates that the actual update timestamp of the target update data exists in the target operation log, and the actual update timestamp is inconsistent with the update time field of the target update data, determine the target update data as non-standard update operation data.

6. The method according to any one of claims 1 to 5, characterized in that, The data of the target table has a unique identifier field; the non-standard operation data is at least two; The data synchronization risk warning according to the non-standard operation data comprises: obtain the total data amount of the non-standard operation data; based on the unique identifier field, remove duplicate data from at least two non-standard operation data to obtain deduplicated operation data, and determine the number of deduplicated operation data as a deduplicated data amount; perform data synchronization risk warning according to the table name of the target table, the total data amount and the deduplicated data amount.

7. The method according to any one of claims 1 to 5, characterized in that, After the method of filtering the data corresponding to the target operation from the data of the target table according to the target operation log to obtain target change data, the method further comprises: perform time comparison according to the creation time field and the update time field of the target change data; if the creation time field of the target change data is greater than the update time field, determine the target change data as the non-standard operation data.

8. A data synchronization risk identification apparatus, characterized by, The device comprises: a target table determination module for determining a target table in a target database; wherein the target table has a creation time field and an update time field, and the target table performs incremental data synchronization based on a preset data synchronization time interval; a time range calculation module for determining a predicted incremental data synchronization time range of the target table according to the time of the last data synchronization of the target table and the data synchronization time interval; an operation log acquisition module for acquiring an operation log corresponding to the target table to obtain a candidate operation log; an operation log filtering module for filtering the candidate operation log within the predicted incremental data synchronization time range according to a target operation to obtain a target operation log; wherein the target operation represents an operation of changing data; a table data filtering module for filtering the data corresponding to the target operation from the data of the target table according to the target operation log to obtain target change data; a time comparison module, configured to perform time comparison according to at least one of the creation time field and the update time field of the target change data and the predicted incremental data synchronization time range, to determine non-standard operation data from the target change data; a risk warning module, configured to perform data synchronization risk warning according to the non-standard operation data.

9. An electronic device, comprising: The electronic device comprises a memory and a processor, the memory stores a computer program, and the processor implements the data synchronization risk identification method in any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium storing a computer program, the computer program comprising instructions that, when executed by a computer, cause the computer to perform the method of any one of claims 1 to 9. The computer program is executed by the processor to implement the data synchronization risk identification method in any one of claims 1 to 7. The computer program is executed by the processor to implement the data synchronization risk identification method in any one of claims 1 to 7.

Images