Data synchronization anomaly detection method and device, equipment and medium

Abstract

The invention relates to the technical field of big data, and discloses a data synchronization anomaly detection method, device, equipment and medium, comprising: acquiring metadata of a source end in a data synchronization link, the metadata of the source end having different hierarchical structures, the different hierarchical structures corresponding to the coverage of different metadata in a database, the method comprises the following steps of: performing anomaly detection on metadata of a source end by utilizing a pre-configured anomaly detection logic before triggering a data synchronization instruction, and performing anomaly detection on the number of elements of a next hierarchical structure according to a dependency relationship among hierarchical structures under the condition that the metadata of the current hierarchical structure does not have anomaly, and under the condition that the metadata of the current hierarchical structure is abnormal, determining abnormal metadata of the source end according to the dependency relationship between the hierarchical structures so as to trigger an abnormal alarm through the abnormal metadata. The data security of a source end can be ensured, and the stability of data synchronization is improved.

Description

Technical Field

[0001] The present invention relates to the fields of big data technology and data processing technology, and in particular to a method, device, equipment and medium for detecting anomalies in data synchronization. Background Art

[0002] In the big data development system, data synchronization is a very important step in data development. It is committed to ensuring that data is migrated efficiently and accurately between different data sources. The stability of data synchronization is related to the stability of the entire data system.

[0003] In the related art, the anomaly detection of the data synchronization process is mainly through post-event monitoring, and manual intervention is performed when an anomaly occurs in the data synchronization task. However, considering that the data synchronization process has very complex business scenarios, in some larger data systems, post-event monitoring is difficult to respond to anomalies that occur during the data synchronization process in a timely manner, which prolongs the data synchronization cycle, affects the data synchronization process, and increases the instability of data synchronization to a certain extent. Summary of the invention

[0004] The present invention provides an artificial intelligence data synchronization anomaly detection method, device, computer equipment and medium to solve the technical problem that it is difficult for post-monitoring to respond in time to anomalies occurring during the data synchronization process, which prolongs the data synchronization cycle and affects the data synchronization process.

[0005] In a first aspect, a method for detecting anomalies in data synchronization is provided, comprising:

[0006] Obtain metadata of a source end in a data synchronization link, wherein the metadata of the source end has different hierarchical structures, the different hierarchical structures correspond to coverage of different metadata in a database, and the hierarchical structures have dependencies;

[0007] Before triggering the data synchronization instruction, using a pre-configured anomaly detection logic to perform anomaly detection on the metadata of the source end;

[0008] When there is no abnormality in the metadata of the current hierarchical structure, an abnormality detection is performed on the metadata of the next hierarchical structure according to the dependency relationship between the hierarchical structures;

[0009] When there is an abnormality in the metadata of the current hierarchical structure, the abnormal metadata of the source end is determined according to the dependency relationship between the hierarchical structures, so as to trigger an abnormality alarm through the abnormal metadata.

[0010] In a second aspect, a data synchronization anomaly detection device is provided, comprising:

[0011] An acquisition module, used to acquire metadata of a source end in a data synchronization link, wherein the metadata of the source end has different hierarchical structures, wherein the different hierarchical structures correspond to coverage of different metadata in a database, and wherein the hierarchical structures have dependencies;

[0012] An anomaly detection module, used to perform anomaly detection on the metadata of the source end using a pre-configured anomaly detection logic before triggering a data synchronization instruction;

[0013] The anomaly detection module is further used to perform anomaly detection on the metadata of the next hierarchical structure according to the dependency relationship between the hierarchical structures when there is no anomaly in the metadata of the current hierarchical structure;

[0014] The abnormality determination module is used to determine the abnormal metadata of the source end according to the dependency relationship between the hierarchical structures when there is an abnormality in the metadata of the current hierarchical structure, so as to trigger an abnormality alarm through the abnormal metadata.

[0015] In a third aspect, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above-mentioned data synchronization anomaly detection method when executing the computer program.

[0016] In a fourth aspect, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the above-mentioned data synchronization anomaly detection method are implemented.

[0017] In the scheme implemented by the above-mentioned data synchronization anomaly detection method, device, computer equipment and storage medium, the metadata of the source end in the data synchronization link can be obtained through the client. The metadata of the source end has different hierarchical structures. Different hierarchical structures correspond to the coverage of different metadata in the database. There is a dependency relationship between the hierarchical structures. Before triggering the data synchronization instruction, the metadata of the source end is detected for anomalies using the pre-configured anomaly detection logic. When there is no anomaly in the metadata of the current hierarchical structure, the metadata of the next hierarchical structure is detected for anomalies according to the dependency relationship between the hierarchical structures. When there is an anomaly in the metadata of the current hierarchical structure, the abnormal metadata of the source end is determined according to the dependency relationship between the hierarchical structures, so as to trigger an abnormal alarm through the abnormal metadata and feed the abnormal alarm back to the client. In the present invention, the abnormal monitoring of data synchronization is advanced to the stage before data synchronization, the abnormal monitoring is locked to the metadata of the source end, and the metadata of the source end is detected for anomalies in a multi-level structure, which can ensure the data security of the source end, so as to warn and intervene in the source end data when anomalies break out, shorten the data synchronization cycle, avoid the occurrence of anomalies in the subsequent data synchronization process, and improve the stability of data synchronization. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.

[0019] Figure 1 It is a schematic diagram of an application environment of a method for detecting anomalies in data synchronization according to an embodiment of the present invention;

[0020] Figure 2 It is a flow chart of a method for detecting anomalies in data synchronization according to an embodiment of the present invention;

[0021] Figure 3 yes Figure 1 A schematic flow chart of a specific implementation of step S10;

[0022] Figure 4 is a flow chart of a method for detecting anomalies in data synchronization according to another embodiment of the present invention;

[0023] Figure 5 yes Figure 4 A schematic flow chart of a specific implementation of step S40;

[0024] Figure 6 yes Figure 1 A schematic flow chart of a specific implementation of step S20;

[0025] Figure 7 It is a structural schematic diagram of a data synchronization anomaly detection device in one embodiment of the present invention;

[0026] Figure 8 is a schematic diagram of a structure of a computer device in one embodiment of the present invention;

[0027] Fig. 9 It is another structural schematic diagram of a computer device in one embodiment of the present invention. DETAILED DESCRIPTION

[0028] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0029] The data synchronization anomaly detection method provided by the embodiment of the present invention can be applied in the following aspects: Figure 1In an application environment, the client communicates with the server through a network. The server can obtain metadata of the source end in the data synchronization link through the client. The metadata of the source end has different hierarchical structures. Different hierarchical structures correspond to the coverage of different metadata in the database. There is a dependency relationship between the hierarchical structures. Before triggering the data synchronization instruction, the metadata of the source end is detected for anomalies using a pre-configured anomaly detection logic. When there is no anomaly in the metadata of the current hierarchical structure, the metadata of the next hierarchical structure is detected for anomalies according to the dependency relationship between the hierarchical structures. When there is an anomaly in the metadata of the current hierarchical structure, the abnormal metadata of the source end is determined according to the dependency relationship between the hierarchical structures, so as to trigger an abnormal alarm through the abnormal metadata and feed the abnormal alarm back to the client. In the present invention, the abnormal monitoring of data synchronization is advanced to the stage before data synchronization, the abnormal monitoring is locked to the metadata of the source end, and the metadata of the source end is detected for anomalies in a multi-level structure, which can ensure the data security of the source end, so as to warn and intervene in the source end data when anomalies break out, shorten the data synchronization cycle, avoid the occurrence of anomalies in the subsequent data synchronization process, and improve the stability of data synchronization. The client may be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, and portable wearable devices. The server may be implemented by an independent server or a server cluster consisting of multiple servers. The present invention is described in detail below through specific embodiments.

[0030] See also Figure 2 As shown, Figure 2 A flow chart of a method for detecting anomalies in data synchronization provided by an embodiment of the present invention includes the following steps:

[0031] S10: Obtain metadata of the source end in the data synchronization link.

[0032] The data synchronization anomaly detection method provided by the present invention can be applied to data processing platforms in various application scenarios or business platforms with data synchronization requirements. The business platform is usually implemented through a server, which can obtain the metadata of the source end in the data synchronization link in real time. Within the enterprise, a large amount of data transmission and synchronization must be carried out between different departments to achieve data sharing and collaboration. For example, the sales department needs to know the product inventory status in a timely manner, and the financial department needs to keep abreast of sales and account balances. To achieve this data synchronization, a big data transmission solution can be used.

[0033] Among them, the data synchronization link is a physical line formed by the synchronization of data from one data node to another data node. There is no switching node in the middle of the line. The data node here can be a database, cluster, server, etc. The source end in the data synchronization link is the data node located at the end of the link. If the data synchronization link is to synchronize the data in database A to database B, the source end in the data synchronization link is database A and database B. Correspondingly, the metadata of the source end is information that describes the attributes of the data in the source end, which is used to support functions such as indicating storage location, historical data, resource search, file records, etc., which is equivalent to descriptive information of data and information resources, such as the source end information table name, table field type, number of field entries, etc.

[0034] Considering the different contents described by metadata in the source end, the metadata in the source end has different hierarchical structures, and there is a dependency relationship between the hierarchical structures. That is to say, metadata can be divided into different hierarchical structures, wherein the higher the hierarchical structure, the greater the coverage of metadata in the corresponding database of the corresponding hierarchical structure. For example, the hierarchical structure describing metadata can be divided into a database hierarchy, a data table hierarchy, and a data indicator hierarchy. In the database hierarchy, the coverage of metadata includes the content of the entire database, in the data table hierarchy, the coverage of metadata includes the content of the data table in the database, and in the data indicator hierarchy, the coverage of metadata includes the content of the data indicator in the data table. Each hierarchical structure depends on the previous hierarchical structure, that is, the content of the data table depends on the content of the database, and the content of the data indicator depends on the content of the data table.

[0035] It should be understood that, according to the data flow direction in the data synchronization link, the source end here includes the upstream source end and the downstream source end, and the metadata connected to different positions in the data synchronization link are pulled through the upstream source end and the downstream source end respectively. Figure 3 As shown, in step S10, that is, obtaining metadata of the source end in the data synchronization link, the following steps are included:

[0036] S11: According to the data flow direction in the data synchronization link, an upstream source end and a downstream source end are located in the data synchronization link.

[0037] S12: According to the upstream source end, metadata with different coverage ranges is periodically obtained from a source database connected to the upstream source end to obtain metadata of the upstream source end.

[0038] S13: According to the downstream source end, metadata with different coverage ranges is periodically acquired from a target database connected to the downstream source end to obtain metadata of the downstream source end.

[0039] For steps S11-S13, considering that the source business scenarios of data synchronization are often very complex, in order to avoid the occurrence of abnormal events during data synchronization, before triggering the data synchronization instruction, since the metadata of the source describes the data information of the start and end of synchronization, by collecting the metadata of the source to perform abnormal monitoring, early warning of data synchronization can be given to ensure the stability of data synchronization.

[0040] Specifically, the upstream source end is the source end of data transmission in the data synchronization link, that is, the upstream source end is connected to the source database of data synchronization. Accordingly, before data synchronization, metadata with different coverage ranges is regularly obtained from the source database connected to the upstream source end to obtain the metadata of the upstream source end.

[0041] Similarly, the downstream source end is the target end of data transmission in the data synchronization link, that is, the target database to which the downstream source end is connected for data synchronization. Accordingly, before data synchronization, metadata with different coverage ranges is regularly obtained from the target database to which the downstream source end is connected to obtain the metadata of the downstream source end.

[0042] It is understandable that once an exception occurs in the source database of the upstream source, the corresponding data on the downstream source will also become abnormal, leading to the failure of the data synchronization task. For example, the database connection of the upstream source changes, resulting in a connection exception in the synchronization task of the downstream source; the table structure or table permissions of the upstream source change, resulting in an exception when synchronizing the table at the downstream source; the data indicators of the upstream source are abnormal, and there are illegal data changes, resulting in logical abnormalities in the established data in the downstream source.

[0043] S20: Before triggering the data synchronization instruction, perform anomaly detection on the metadata of the source end using a pre-configured anomaly detection logic.

[0044] Among them, the pre-configured anomaly detection logic is usually set according to the metadata of different hierarchical structures, taking into account that the metadata of different hierarchical structures realizes the data synchronization process through different structural features. For example, the metadata of the database hierarchical structure needs to be connected to realize data synchronization, the metadata of the data table hierarchical structure realizes data synchronization through attribute features such as table fields, and the metadata of the data indicator hierarchical structure realizes data synchronization through data types.

[0045] Accordingly, before step S20, Figure 4 As shown, the above data synchronization anomaly detection method also includes the following steps:

[0046] S50. According to the structural features of the source metadata at different hierarchical structures, anomaly detection logic applicable to different hierarchical structures is set.

[0047] It should be noted that different hierarchical structures include at least a first hierarchical structure, a second hierarchical structure, and a third hierarchical structure. Since the metadata between different hierarchical structures are dependent, that is, the metadata of the first hierarchical structure is the conditional basis for executing the data synchronization task. Only when the metadata of the first hierarchical structure is available, the metadata of the second hierarchical structure has the condition for executing the data synchronization task. Only when the metadata of the second hierarchical structure is available, the metadata of the third hierarchical structure has the condition for executing the data synchronization task. In some embodiments of the invention, such as Figure 5 As shown, a specific solution for setting anomaly detection logic applicable to different hierarchical structures is provided. In S40, that is, according to the structural features of the metadata of the source end at different hierarchical structures, anomaly detection logic applicable to different hierarchical structures is set, which specifically includes the following steps S51-S53;

[0048] S51. According to the availability of the database connection of the metadata of the source end on the first hierarchical structure, set the anomaly detection logic of the first hierarchical structure.

[0049] S52: according to the consistency of the table fields of the metadata of the source end on the second hierarchical structure, set the anomaly detection logic of the second hierarchical structure.

[0050] S53: according to the risk of the data indicator of the metadata of the source end in the third-level structure, set the anomaly detection logic of the third-level structure.

[0051] In this embodiment, the first level structure describes the metadata of the source end from the database level. Since the anomaly detection at the database level is to detect whether the database connection is available, the corresponding anomaly detection logic is to detect the connection availability of the metadata at the database level. Specifically, scripting technology can be used to regularly test the connection availability of the upstream source end to the database.

[0052] In this embodiment, the second hierarchical structure describes the metadata of the source end from the data table level. Since the anomaly detection of the data table is to detect whether the table fields are consistent, the corresponding anomaly detection logic is to perform consistency detection of the table fields at the data table level for the metadata. Specifically, the consistency analysis of the table fields can be performed through the table structures corresponding to the upstream source end and the downstream source end.

[0053] In this embodiment, the third-level structure describes the metadata of the source end from the data indicator level. Since the anomaly detection at the data indicator level is to detect whether there is an anomaly in the indicator, the corresponding anomaly detection logic is to perform indicator anomaly detection on the metadata at the data indicator level. Specifically, the consistency analysis of the table fields can be performed through the table structure corresponding to the upstream source end and the downstream source end.

[0054] Accordingly, in some embodiments of the invention, Figure 6 As shown, an anomaly detection solution is provided. In S20, the metadata of the source end is subjected to anomaly detection using a pre-configured anomaly detection logic, which specifically includes the following steps S21-S23;

[0055] S21. Utilize the anomaly detection logic of the first hierarchical structure to traverse and access the database of the upstream source end, so as to perform anomaly detection on the availability of the database connection of the metadata of the source end on the first hierarchical structure according to the traversal access result.

[0056] S22. Using the anomaly detection logic of the second hierarchical structure, compare the table of the upstream source end with the table fields of the downstream source end, so as to perform anomaly detection on the consistency of the table fields of the metadata of the source end on the second hierarchical structure according to the comparison result.

[0057] S23. Using the anomaly detection logic of the third hierarchical structure, the data indicators of the upstream source are checked to perform anomaly detection on the risk of the data indicators of the metadata of the source in the third hierarchical structure according to the checking results.

[0058] Specifically, in the process of performing anomaly detection on the metadata of the source end using the anomaly detection logic of the first-level structure, a script file can be used to traverse and access the database of the upstream source end to determine whether the database of the upstream source end is connected and available. If the connection is available, it is determined that the metadata of the source end has connection availability in the metadata database of the first-level structure. If the connection is unavailable, it is determined that the metadata of the source end has a connection anomaly in the metadata database of the first-level structure.

[0059] Specifically, in the process of using the anomaly detection logic of the second-level structure to detect anomalies in the metadata of the source end, the anomaly detection logic of the second-level structure can be used to obtain the table structure of the upstream source end and the table structure of the downstream source end respectively, and compare the table fields of the upstream source end with the table fields of the downstream source end according to the table structure of the upstream source end and the table structure of the downstream source end. If the comparison result is that the table fields are consistent, it is determined that the metadata of the source end has consistency in the table fields of the metadata on the second-level structure. If the comparison result is that the table fields are inconsistent, it is determined that the metadata of the source end has a field anomaly in the table structure of the metadata on the second-level structure. For example, a field change or a permission change in the table structure of the upstream source end will cause the table structure of the upstream source end and the downstream source end to be inconsistent, affecting the execution of subsequent data synchronization tasks.

[0060] Specifically, in the process of using the anomaly detection logic of the third-level structure to detect anomalies in the metadata of the source end, the anomaly detection logic of the third-level structure can be used to set anomaly indicator conditions for monitoring key fields, and the key fields of the upstream source end can be checked according to the anomaly indicator conditions. If the key field does not meet the anomaly indicator conditions, it is determined that the data indicators of the metadata of the source end on the third-level structure are safe. If the key field meets the anomaly indicator conditions, it is determined that the metadata of the source end has an indicator anomaly in the key field of the metadata on the third-level structure. Here, the anomaly indicator condition can be the value of the key field in the upstream source end or the component of the key field. For example, the key field in the upstream source end is 0 or the field is empty, or the key field of the upstream source end has a duplicate business component, or the business component of the key field in the upstream source end triggers a system anomaly, etc. If the key field of the upstream source end is detected.

[0061] It can be seen from the above scheme that before triggering the data synchronization instruction, the anomaly detection is locked to the source end, and a variety of detection methods are combined to perform anomaly detection of different hierarchical structures on the metadata of the source end, thereby improving the anomaly detection range of data synchronization. It can warn and intervene to eliminate abnormal events of data synchronization before problems break out, thereby ensuring the stability of the data synchronization system to the greatest extent.

[0062] S30: When there is no abnormality in the metadata of the current hierarchical structure, perform abnormality detection on the metadata of the next hierarchical structure according to the dependency relationship between the hierarchical structures.

[0063] S40: When there is an abnormality in the metadata of the current hierarchical structure, determine the abnormal metadata of the source end according to the dependency relationship between the hierarchical structures, so as to trigger an abnormality alarm through the abnormal metadata.

[0064] It is understandable that, considering that there is a dependency between the hierarchical structures, the dependency is that an abnormality in the metadata of the current hierarchical structure will cause the metadata of the next hierarchical structure to be unable to execute the data synchronization task, while an abnormality in the metadata of the current hierarchical structure will not affect the execution of the data synchronization task by the metadata of the previous structure. In other words, an abnormality in the metadata of the first hierarchical structure will cause an abnormality in the metadata of the second hierarchical structure, and the data synchronization task cannot be executed, while an abnormality in the metadata of the second hierarchical structure will not affect the execution of the data synchronization task by the metadata of the first hierarchical structure.

[0065] It can be seen that in the above scheme, by performing anomaly detection on the metadata of the source at different levels, the anomaly detection of data synchronization can be locked on the source. Based on the detection of the metadata combined with multi-level structures, the anomaly detection scheme can be effectively deployed before data synchronization, and early warning of abnormal events can be given, thereby reducing data synchronization anomalies caused by source problems and ensuring the stability of the data synchronization process.

[0066] It should be understood that the order of execution of the steps in the above embodiment does not necessarily mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present invention.

[0067] In one embodiment, a data synchronization anomaly detection device is provided, and the data synchronization anomaly detection device corresponds to the data synchronization anomaly detection method in the above embodiment. Figure 7 As shown, the data synchronization anomaly detection device includes an acquisition module 101, an anomaly detection module 102 and an anomaly determination module 103. Each functional module is described in detail as follows:

[0068] The acquisition module 101 is used to acquire metadata of a source end in a data synchronization link, wherein the metadata of the source end has different hierarchical structures, and the different hierarchical structures correspond to coverage of different metadata in a database, and the hierarchical structures have dependencies;

[0069] The anomaly detection module 102 is used to perform anomaly detection on the metadata of the source end using a pre-configured anomaly detection logic before triggering the data synchronization instruction;

[0070] The anomaly detection module 102 is further used to perform anomaly detection on the metadata of the next hierarchical structure according to the dependency relationship between the hierarchical structures when there is no anomaly in the metadata of the current hierarchical structure;

[0071] The abnormality determination module 103 is used to determine the abnormal metadata of the source end according to the dependency relationship between the hierarchical structures when there is an abnormality in the metadata of the current hierarchical structure, so as to trigger an abnormality alarm through the abnormal metadata.

[0072] In one embodiment, the determination module 101 is specifically configured to:

[0073] According to the data flow direction in the data synchronization link, locating the upstream source end and the downstream source end in the data synchronization link;

[0074] According to the upstream source end, metadata with different coverage ranges is periodically acquired from a source database connected to the upstream source end to obtain metadata of the upstream source end;

[0075] According to the downstream source end, metadata with different coverage ranges is periodically acquired from a target database connected to the downstream source end to obtain metadata of the downstream source end.

[0076] In one embodiment, the apparatus further comprises:

[0077] The setting module is used to set anomaly detection logic applicable to different hierarchical structures according to structural features of the metadata of the source end at different hierarchical structures before performing anomaly detection on the metadata of the source end using the pre-configured anomaly detection logic.

[0078] In one embodiment, the different layer structures include a first layer structure, a second layer structure and a third layer structure, and the setting module is further used for:

[0079] According to the availability of the database connection of the metadata of the source end on the first level structure, setting the anomaly detection logic of the first level structure;

[0080] According to the consistency of the table fields of the metadata of the source end on the second hierarchical structure, setting the anomaly detection logic of the second hierarchical structure;

[0081] According to the risk of the data indicator of the metadata of the source end in the third-level structure, an anomaly detection logic of the third-level structure is set.

[0082] In one embodiment, the anomaly detection module 102 is specifically configured to:

[0083] Using the anomaly detection logic of the first hierarchical structure, traversing and accessing the database of the upstream source end, so as to perform anomaly detection on the availability of the database connection of the metadata of the source end on the first hierarchical structure according to the traversal access result;

[0084] Using the anomaly detection logic of the second hierarchical structure, the table of the upstream source end is compared with the table fields of the downstream source end, so as to perform anomaly detection on the consistency of the table fields of the metadata of the source end on the second hierarchical structure according to the comparison result;

[0085] The data indicators of the upstream source are checked by utilizing the anomaly detection logic of the third-level structure, so as to perform anomaly detection on the risk of the data indicators of the metadata of the source on the third-level structure according to the checking result.

[0086] In one embodiment, the anomaly detection module 102 is further configured to:

[0087] Using the anomaly detection logic of the second hierarchical structure, respectively obtain the table structure of the upstream source end and the table structure of the downstream source end;

[0088] According to the table structure of the upstream source end and the table structure of the downstream source end, comparing the table fields of the upstream source end with the table fields of the downstream source end;

[0089] If the comparison result is that the table fields are consistent, it is determined that the table fields of the metadata of the source end on the second level structure are consistent;

[0090] If the comparison result is that the table fields are inconsistent, it is determined that there is an abnormality in the table fields of the metadata of the source end in the second hierarchical structure.

[0091] In one embodiment, the anomaly detection module 102 is further configured to:

[0092] Using the anomaly detection logic of the third hierarchical structure, setting an abnormal indicator condition for monitoring key fields;

[0093] Check the key fields of the upstream source end according to the abnormal indicator conditions;

[0094] If the key field does not meet the abnormal index condition, it is determined that the data index of the metadata of the source end on the third hierarchical structure is safe;

[0095] If the key field meets the abnormal index condition, it is determined that there is an abnormality in the data index of the metadata of the source end in the third hierarchical structure.

[0096] The present invention provides an anomaly detection device for data synchronization, which advances the anomaly monitoring of data synchronization to the stage before data synchronization, locks the anomaly monitoring to the metadata of the source end, and performs multi-level structure anomaly detection on the metadata of the source end, which can ensure the data security of the source end, so as to give early warning and intervention to the source end data when anomalies break out, shorten the data synchronization cycle, avoid the occurrence of anomalies in subsequent data synchronization processes, and improve the stability of data synchronization.

[0097] For the specific definition of the data synchronization anomaly detection device, please refer to the definition of the intelligent question and answer method above, which will not be repeated here. Each module in the above-mentioned data synchronization anomaly detection device can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0098] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 8As shown. The computer device includes a processor, a memory, a network interface and a database connected via a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile and / or volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external client via a network connection. When the computer program is executed by the processor, it realizes the functions or steps of a data synchronization anomaly detection method server side.

[0099] In one embodiment, a computer device is provided. The computer device may be a client, and its internal structure diagram may be as follows: Fig. 9 As shown. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external server via a network connection. When the computer program is executed by the processor, it realizes the functions or steps of a client side of a data synchronization anomaly detection method

[0100] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the following steps when executing the computer program:

[0101] Obtain metadata of a source end in a data synchronization link, wherein the metadata of the source end has different hierarchical structures, the different hierarchical structures correspond to coverage of different metadata in a database, and the hierarchical structures have dependencies;

[0102] Before triggering the data synchronization instruction, using a pre-configured anomaly detection logic to perform anomaly detection on the metadata of the source end;

[0103] When there is no abnormality in the metadata of the current hierarchical structure, an abnormality detection is performed on the metadata of the next hierarchical structure according to the dependency relationship between the hierarchical structures;

[0104] When there is an abnormality in the metadata of the current hierarchical structure, the abnormal metadata of the source end is determined according to the dependency relationship between the hierarchical structures, so as to trigger an abnormality alarm through the abnormal metadata.

[0105] In one embodiment, a computer readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0106] Obtain metadata of a source end in a data synchronization link, wherein the metadata of the source end has different hierarchical structures, the different hierarchical structures correspond to coverage of different metadata in a database, and the hierarchical structures have dependencies;

[0107] Before triggering the data synchronization instruction, using a pre-configured anomaly detection logic to perform anomaly detection on the metadata of the source end;

[0108] When there is no abnormality in the metadata of the current hierarchical structure, an abnormality detection is performed on the metadata of the next hierarchical structure according to the dependency relationship between the hierarchical structures;

[0109] When there is an abnormality in the metadata of the current hierarchical structure, the abnormal metadata of the source end is determined according to the dependency relationship between the hierarchical structures, so as to trigger an abnormality alarm through the abnormal metadata.

[0110] It should be noted that the above functions or steps that can be implemented by the computer-readable storage medium or computer device can refer to the relevant descriptions on the server side and the client side in the aforementioned method embodiment. To avoid repetition, they will not be described one by one here.

[0111] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0112] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0113] The embodiments described above are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included in the protection scope of the present invention.

Claims

1. A data synchronization anomaly detection method, characterized in that: include: Obtain metadata of a source end in a data synchronization link, wherein the metadata of the source end has different hierarchical structures, the different hierarchical structures correspond to coverage of different metadata in a database, and the hierarchical structures have dependencies; Before triggering the data synchronization instruction, using a pre-configured anomaly detection logic to perform anomaly detection on the metadata of the source end; When there is no abnormality in the metadata of the current hierarchical structure, an abnormality detection is performed on the metadata of the next hierarchical structure according to the dependency relationship between the hierarchical structures; When there is an abnormality in the metadata of the current hierarchical structure, the abnormal metadata of the source end is determined according to the dependency relationship between the hierarchical structures, so as to trigger an abnormality alarm through the abnormal metadata.

2. The data synchronization anomaly detection method according to claim 1, characterized in that: The step of obtaining metadata of a source end on a data synchronization link includes: According to the data flow direction in the data synchronization link, locating the upstream source end and the downstream source end in the data synchronization link; According to the upstream source end, metadata with different coverage ranges is periodically obtained from a source database connected to the upstream source end to obtain metadata of the upstream source end; According to the downstream source end, metadata with different coverage ranges is periodically acquired from a target database connected to the downstream source end to obtain metadata of the downstream source end.

3. The data synchronization anomaly detection method according to claim 1, characterized in that: Before performing anomaly detection on the metadata of the source end using the pre-configured anomaly detection logic, the method further includes: According to the structural features of the metadata of the source end at different hierarchical structures, anomaly detection logic applicable to different hierarchical structures is set.

4. The data synchronization anomaly detection method according to claim 3, characterized in that: The different hierarchical structures include a first hierarchical structure, a second hierarchical structure, and a third hierarchical structure. The abnormality detection logic applicable to the different hierarchical structures is set according to the structural features of the metadata of the source end at the different hierarchical structures, including: According to the availability of the database connection of the metadata of the source end on the first level structure, setting the anomaly detection logic of the first level structure; According to the consistency of the table fields of the metadata of the source end on the second hierarchical structure, setting the anomaly detection logic of the second hierarchical structure; According to the risk of the data indicator of the metadata of the source end in the third-level structure, an abnormality detection logic of the third-level structure is set.

5. The data synchronization anomaly detection method according to claim 4, characterized in that: The using of a pre-configured anomaly detection logic to perform anomaly detection on the metadata of the source end includes: Using the anomaly detection logic of the first hierarchical structure, traversing and accessing the database of the upstream source end, so as to perform anomaly detection on the availability of the database connection of the metadata of the source end on the first hierarchical structure according to the traversal access result; Using the anomaly detection logic of the second hierarchical structure, the table of the upstream source end is compared with the table fields of the downstream source end, so as to perform anomaly detection on the consistency of the table fields of the metadata of the source end on the second hierarchical structure according to the comparison result; The data indicators of the upstream source are checked by utilizing the anomaly detection logic of the third-level structure, so as to perform anomaly detection on the risk of the data indicators of the metadata of the source on the third-level structure according to the checking result.

6. The data synchronization anomaly detection method according to claim 5, characterized in that: The using the anomaly detection logic of the second hierarchical structure to compare the table of the upstream source end with the table of the downstream source end, so as to perform anomaly detection on the consistency of the fields of the metadata of the source end on the second hierarchical structure according to the comparison result, includes: Using the anomaly detection logic of the second hierarchical structure, respectively obtain the table structure of the upstream source end and the table structure of the downstream source end; According to the table structure of the upstream source end and the table structure of the downstream source end, comparing the table fields of the upstream source end with the table fields of the downstream source end; If the comparison result is that the table fields are consistent, it is determined that the table fields of the metadata of the source end on the second level structure are consistent; If the comparison result is that the table fields are inconsistent, it is determined that there is an abnormality in the table fields of the metadata of the source end in the second hierarchical structure.

7. The data synchronization anomaly detection method according to claim 5, characterized in that: The method of using the anomaly detection logic of the third hierarchical structure to check the data indicators of the upstream source end, and performing anomaly detection on the risk of the data indicators of the metadata of the source end on the third hierarchical structure according to the checking result, includes: Using the anomaly detection logic of the third hierarchical structure, setting an abnormal indicator condition for monitoring key fields; Check the key fields of the upstream source end according to the abnormal indicator conditions; If the key field does not meet the abnormal index condition, it is determined that the data index of the metadata of the source end on the third hierarchical structure is safe; If the key field meets the abnormal index condition, it is determined that there is an abnormality in the data index of the metadata of the source end in the third hierarchical structure.

8. A data synchronization anomaly detection device, characterized in that: include: An acquisition module, used to acquire metadata of a source end in a data synchronization link, wherein the metadata of the source end has different hierarchical structures, wherein the different hierarchical structures correspond to coverage of different metadata in a database, and the hierarchical structures have dependencies; An anomaly detection module, used to perform anomaly detection on the metadata of the source end using a pre-configured anomaly detection logic before triggering a data synchronization instruction; The anomaly detection module is further used to perform anomaly detection on the metadata of the next hierarchical structure according to the dependency relationship between the hierarchical structures when there is no anomaly in the metadata of the current hierarchical structure; The abnormality determination module is used to determine the abnormal metadata of the source end according to the dependency relationship between the hierarchical structures when there is an abnormality in the metadata of the current hierarchical structure, so as to trigger an abnormality alarm through the abnormal metadata.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the data synchronization anomaly detection method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the data synchronization anomaly detection method according to any one of claims 1 to 7 are implemented.

Images