Data synchronization method and device, electronic equipment and storage medium
By obtaining monitoring configuration information and operation identifiers, determining the data to be monitored from the first structural data and synchronizing the second structural data, the problems of low data synchronization efficiency and complex architecture in the prior art are solved, and efficient data synchronization and Simplified deployment structure.
Patent Information
- Application Number
- CN202411130133.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-16
- Publication Date
- 2024-12-06
AI Technical Summary
The existing data synchronization method is inefficient and complex in architecture, and it is necessary to deploy timers and data acquisition tools in the first structural data to synchronize the second structural data, resulting in increased resource consumption and implementation difficulty.
By acquiring the monitoring data identification and operation identification in the monitoring configuration information, the data to be monitored is determined from the first structure data, and the data to be synchronized from the second structure data based on the operation results, data synchronization processing is realized, and the first The requirement for deploying complex processing tools in structured data.
The data synchronization efficiency is improved, the deployment structure of the synchronization solution is simplified, real-time synchronization between the first structural data and the second structural data is realized, and resource consumption and implementation complexity is reduced.
Smart Images

Figure CN119088880A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of data processing technology, and in particular to a data synchronization method, device, electronic device and storage medium. Background Art
[0002] In the field of data processing, business scenarios such as data storage and data query are often involved. For business systems with huge amounts of data, in order to increase the data query rate, the data generated by the business system is generally stored by combining two different structures of data, such as relational data and graph data. Among them, relational data is used to produce business data, and graph data is used to store business data in a graph-theoretic manner based on data relationships.
[0003] However, when storing data in two different data structures, in order to maintain the consistency of business data in the two data structures, it is generally necessary to synchronize the business data from the first structure data to the second structure data. The existing method of data synchronization is generally to configure a timer and data acquisition and other processing tools in the first structure data, configure the timer according to the scheduling time interval, and regularly query the changes of business data in the first structure data. When the business data in the first structure data is detected to have changed, the data acquisition tool is started to collect the changed data in the first structure data, and then the corresponding data in the second structure data is synchronously changed accordingly.
[0004] The above-mentioned method of data synchronization has problems such as slow data synchronization efficiency and complex solution implementation architecture. Summary of the invention
[0005] The present application provides a data synchronization method, device, electronic device and storage medium, which can improve data synchronization efficiency and simplify the deployment structure of the data synchronization solution.
[0006] In a first aspect, the present application provides a data synchronization method, comprising:
[0007] Acquire monitoring configuration information, the monitoring configuration information including a monitoring data identifier and a monitoring operation identifier; determine the data to be monitored from the first structure data according to the monitoring data identifier, and obtain the operation result of the data to be monitored according to the monitoring operation identifier; determine the data to be synchronized corresponding to the data to be monitored from the second structure data; and synchronize the data to be synchronized according to the operation result.
[0008] Optionally, determining the data to be monitored from the first structure data according to the monitoring data identifier includes: acquiring at least one class name information in the first structure data; and determining the data to be monitored according to the monitoring data identifier and at least one class name information.
[0009] Optionally, determining the data to be monitored based on the monitoring data identifier and at least one of the class name information includes: for a current class name information in at least one of the class name information, obtaining a current compilation identifier obtained after performing a compilation operation on the current class name information; when the current compilation identifier and the monitoring data identifier are consistent, determining the data to be monitored based on the current compilation identifier; when the current compilation identifier and the monitoring data identifier are inconsistent, performing an operation of compiling the next class name information.
[0010] Optionally, obtaining the operation result of the data to be monitored according to the monitoring operation identifier includes: monitoring the data to be monitored through a set data monitoring function; and obtaining the operation result when the operation corresponding to the monitoring operation identifier is monitored on the data to be monitored.
[0011] Optionally, determining the data to be synchronized corresponding to the data to be monitored from the second structure data includes: acquiring a first structure language corresponding to the data to be monitored; mapping the first structure language to obtain a second structure language corresponding to the data to be monitored; and determining the data to be synchronized from the second structure data according to the second structure language.
[0012] Optionally, the monitoring operation identifier includes a data addition identifier and / or a data deletion identifier, and correspondingly, the operation result includes a data addition result and / or a data deletion result. After synchronizing the data to be synchronized according to the operation result, it also includes: determining the operation parameter identifier of the first structure data according to the operation result to obtain a first operation parameter identifier; determining the operation parameter identifier of the second structure data according to the synchronization processing result to obtain a second operation parameter identifier; and performing data synchronization verification according to the first operation parameter identifier and the second operation parameter identifier.
[0013] Optionally, the monitoring operation identifier includes a data update identifier, and correspondingly, the operation result includes a data update result. After synchronizing the data to be synchronized according to the operation result, it also includes: determining the update time of the first structure data according to the data update result to obtain a first timestamp; determining the update time of the second structure data according to the data synchronization processing result to obtain a second timestamp; and performing data synchronization verification according to the first timestamp and the second timestamp.
[0014] In a second aspect, the present application provides a data synchronization device, the device comprising:
[0015] An information acquisition module, used to acquire monitoring configuration information, wherein the monitoring configuration information includes a monitoring data identifier and a monitoring operation identifier;
[0016] A result acquisition module, used to determine the data to be monitored from the first structure data according to the monitoring data identifier, and to acquire the operation result of the data to be monitored according to the monitoring operation identifier;
[0017] A data determination module, used to determine the data to be synchronized corresponding to the data to be monitored from the second structure data;
[0018] A synchronization processing module is used to perform synchronization processing on the data to be synchronized according to the operation result.
[0019] In a third aspect, the present application further provides an electronic device, the electronic device comprising:
[0020] at least one processor; and
[0021] a memory communicatively connected to the at least one processor; wherein,
[0022] The memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the data synchronization method described in any embodiment of the present application.
[0023] In a fourth aspect, the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the data synchronization method described in any embodiment of the present application when executed.
[0024] In a fifth aspect, the present application also provides a computer program product, including a computer program, which, when executed by a processor, implements the data synchronization method described in any embodiment of the present application.
[0025] The data synchronization solution provided by the embodiment of the present application configures the monitoring configuration information locally in advance, so that the monitoring data identifier and the monitoring operation identifier contained in the monitoring configuration information can be obtained, thereby determining the data to be monitored from the first structure data through the monitoring data identifier, and then obtaining the operation result of the data to be monitored through the monitoring operation identifier, and then determining the data to be synchronized corresponding to the data to be monitored from the second structure data, and finally synchronizing the data to be synchronized according to the operation result, so as to achieve the purpose of keeping the data content in the first structure data and the second structure data consistent. The solution provided by this embodiment does not need to deploy complex processing tools in the first structure data, and can achieve the synchronization of the first structure data and the second structure data through the monitoring configuration information, which has the beneficial effect of improving the efficiency of data synchronization and simplifying the deployment structure of the solution.
[0026] It should be noted that the above computer instructions may be stored in whole or in part on a computer-readable storage medium, wherein the computer-readable storage medium may be packaged together with the processor of the data synchronization device, or may be packaged separately from the processor of the data synchronization device, which is not limited in this application.
[0027] The description of the second, third, fourth and fifth aspects of the present application can refer to the detailed description of the first aspect; and the beneficial effects of the description of the second, third, fourth and fifth aspects can refer to the beneficial effect analysis of the first aspect, which will not be repeated here.
[0028] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present application, nor is it intended to limit the scope of the present application. Other features of the present application will become easily understood through the following description.
[0029] It is understandable that before using the technical solutions disclosed in the embodiments of this application, the type, scope of use, and usage scenarios of the personal information involved in this application should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations. BRIEF DESCRIPTION OF THE DRAWINGS
[0030] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.
[0031] Figure 1 It is a flowchart of a data synchronization method provided by an embodiment of the present application;
[0032] Figure 2 is a structural diagram of a data synchronization device provided in an embodiment of the present application;
[0033] Figure 3 It is a structural diagram of an electronic device of a data synchronization method provided in an embodiment of the present application. DETAILED DESCRIPTION
[0034] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the present application will be clearly and completely described below in conjunction with the drawings in the present embodiment. Obviously, the described embodiment is only a part of the embodiment of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without creative work should fall within the scope of protection of the present application.
[0035] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.
[0036] The present application will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the present application, rather than to limit the present application. It should also be noted that, for ease of description, only the parts related to the present application, rather than all structures, are shown in the accompanying drawings.
[0037] Figure 1 A flow chart of a data synchronization method provided in an embodiment of the present application, this embodiment can be applied to the case where, when business data in a business system changes, second structure data associated with the first structure data is synchronized according to the change mode of the first structure data. The method can be executed by a data synchronization device, which can be implemented in the form of hardware and / or software and integrated in an electronic device that executes the method. Preferably, the electronic device in the embodiment of the present application can be a server, or a computer device, etc.
[0038] Before introducing the data synchronization solution provided by the embodiment of the present application, the existing data synchronization solution is first described as follows. When the existing solution uses a timer to query the changes of business data in the first structure data at preset intervals, when it is detected that the business data has changed in the first structure data, the data collection tool is started to collect the changed data in the first structure data. This method of determining the frequency of data synchronization by a timer cannot achieve real-time collection, and the startup of the data collection tool also takes some time, resulting in insufficient timeliness of data collection; and the existing data synchronization solution requires the deployment and configuration of timers, data collection tools, and data processing programs to complete data synchronization and conversion. In actual cluster deployment, timers, data collection tools, and data processing programs need to be deployed on corresponding servers to ensure cluster stability. The existing implementation scheme requires more resources to be deployed during implementation, resulting in a more complex structure.
[0039] In view of this, the data synchronization solution provided in this embodiment does not require the deployment of collection components and timers, and does not require the modification of the underlying code of the business system. The data synchronization between the first structure data and the second structure data can be achieved through a one-click start operation, thereby overcoming the problems existing in the existing technical solutions.
[0040] Specifically, refer to Figure 1 The data synchronization method of this embodiment includes but is not limited to the following steps:
[0041] S110. Acquire monitoring configuration information, where the monitoring configuration information includes a monitoring data identifier and a monitoring operation identifier.
[0042] Before introducing the data synchronization solution provided by the embodiment of the present application, it is necessary to explain the execution subject of the present embodiment. The execution subject of the embodiment of the present application can be considered as a computer device with a data synchronization plug-in installed. Among them, the data synchronization plug-in can be embodied in the form of an installation package or an agent program, so that when the relevant business system is monitored, the data synchronization plug-in can be dynamically called through the pre-deployed application program interface (Application Program Interface, referred to as API), so that the data synchronization of the second structure data associated with the first structure data can be realized without modifying the underlying code of the business system.
[0043] In this embodiment, the method for obtaining monitoring configuration information can be to pre-establish a monitoring configuration file locally, and the monitoring configuration file at least contains the data identifier to be monitored, the operation identifier to be monitored, and the second structure data identifier to be synchronized. When the business system is running, after calling the data synchronization plug-in through the API interface, the data synchronization plug-in obtains the corresponding monitoring configuration file from the local to obtain the required monitoring configuration information from the monitoring configuration file. In this embodiment, the monitoring configuration information includes a monitoring data identifier and a monitoring operation identifier. Among them, the monitoring data identifier is the data identifier to be monitored obtained from the monitoring configuration file; the monitoring operation identifier is the operation identifier to be monitored obtained from the monitoring configuration file, etc.
[0044] The business system mentioned in the embodiments of the present application may be a business system used by enterprises and institutions. Generally, during the operation of the business system, corresponding business data will be generated according to the changes in the business content. For example, deposit and withdrawal business data and investment and financial management business data in the banking field; population change business data, housing and land business data, and social security payment business data in the government field; user preference business data and precise recommendation business data involved in the field of big data analysis, etc. The content indicated by the specific business data is not limited here.
[0045] Generally, since the business data generated by the business system is massive, in order to facilitate the storage and query of the business data, the data generated by the business system is conventionally stored by combining two data with different structures, which are described as first structure data and second structure data in this embodiment. Exemplarily, taking the first structure data as relational data as an example, the basic business data generated by the business system can be stored through relational data to obtain table data. The method of storing business data through table data has the advantages of high data consistency, good integrity and low redundancy. However, in scenarios involving multi-layer relationship queries, the method of using multi-table association loops to traverse the files corresponding to each table data in the relational database has the problem of slow query performance. Therefore, on the basis of the relational database, it is further possible to combine the second data structure, such as a graph database, to convert the relational data into graph data, so as to reflect the association relationship of the business data through the graph data, which can improve the data query efficiency.
[0046] It should be noted that, in the data synchronization solution provided in this embodiment, in addition to the first structure data being relational data and the second structure data being graph data, the first structure data can also be graph data, and the corresponding second structure data can also be relational data; optionally, the first structure data or the second structure data can also be key-value storage data, etc., and the specific data types corresponding to the first structure data and the second structure data are not limited here.
[0047] In this embodiment, the monitoring data identifier is used to clarify the data to be monitored in the first data structure in subsequent steps. The current monitoring data identifier can be a data name or a data address, etc. Taking the first structure data as relational data as an example, multiple types of business data may be generated during the operation of the business system. Then, by using the monitoring data identifier to clarify the data to be monitored during the operation of the business system, it is helpful to quickly locate the data to be monitored. The reason for this is that in actual data synchronization application scenarios, in order to reduce the workload of data synchronization, it is not necessary to synchronize all data, such as system operation data, user access data, etc. This type of data only needs to retain the corresponding log locally. Therefore, the data to be monitored can be clarified by configuring the monitoring data identifier. The current data to be monitored can be table data, or graph data, etc. The specific data type of the data to be monitored is related to the data type corresponding to the first structure data.
[0048] The monitoring operation identifier is used to monitor the specific operation mode of the monitored data. It is easy to think that the conventional operation modes of data generally include adding operation, deleting operation, modifying (or updating) operation and viewing operation, etc. Among them, the monitoring operation identifier can distinguish each data operation mode by combining numbers, letters and / or special symbols. In the present embodiment, the purpose of setting the monitoring operation identifier is to synchronize the second data structure in a corresponding manner by monitoring the operation mode of the first data structure. In the present embodiment, data synchronization operations are only performed on the data involving substantial changes in the first structure data. Therefore, when the monitoring data identifier is an identifier corresponding to the viewing operation, there is no need to execute the data synchronization scheme provided in the present embodiment.
[0049] S120: Determine the data to be monitored from the first structure data according to the monitoring data identifier, and obtain the operation result of the data to be monitored according to the monitoring operation identifier.
[0050] When determining the data to be monitored from the first structure data according to the monitoring data identifier, taking the monitoring data identifier as a file name as an example, the file to be monitored can be determined from the multiple file names contained in the first structure data according to the file name. Then, obtaining the specific operation result of the monitoring operation identifier on the file to be monitored helps to synchronize the second structure data according to the current operation result. In this embodiment, there is no need to deploy complex processing tools. The synchronization of the first structure data and the second structure data can be achieved through monitoring configuration information, which has the beneficial effect of improving data synchronization efficiency and simplifying the deployment structure of the solution.
[0051] A preferred embodiment, the preferred implementation method of this embodiment of determining the data to be monitored from the first structure data according to the monitoring data identifier includes: obtaining at least one class name information in the first structure data; determining the data to be monitored according to the monitoring data identifier and at least one class name information.
[0052] At least one class name information is included in the computer program running the business system, and the at least one class name information is stored in the data access layer (Data Access Object, DAO) of the business system, and then the data to be monitored is determined by accessing the at least one class name information contained in the DAO layer. Exemplarily, the class name information containing the monitoring data identifier in the at least one class name information is determined as the target class name information, and then the data corresponding to the target class name information is determined as the data to be monitored.
[0053] Specifically, the present application also provides a preferred embodiment, wherein the above-mentioned determination of the data to be monitored according to the monitoring data identifier and at least one class name information can be implemented in the following manner:
[0054] For the current class name information in at least one class name information, obtain the current compilation identifier obtained after performing the compilation operation on the current class name information; when the current compilation identifier and the monitoring data identifier are consistent, determine the data to be monitored according to the current compilation identifier; when the current compilation identifier and the monitoring data identifier are inconsistent, perform the operation of compiling the next class name information. The purpose of performing the compilation operation on the current class name information in this embodiment is to clarify which file the current class name information is for. Therefore, after the current compilation identifier is obtained after performing the compilation operation on the current class name information, if the current compilation identifier is consistent with the monitoring data identifier, it indicates that the data corresponding to the current compilation identifier is the data to be monitored, otherwise perform the operation of compiling the next class name information.
[0055] From the perspective of computer program implementation, taking the implementation of a computer program through a java program as an example, then, after the data to be monitored is determined according to the current compilation identifier, it indicates that the data corresponding to the current compilation identifier needs to be monitored, so the current class name information can be loaded into the javassist ctClass object. The role of the javassist ctClass object is to configure the corresponding program statement according to the monitoring data identifier and the monitoring operation identifier in the monitoring configuration information. Taking the first data structure as relational data as an example, then after the current class name information is loaded into the javassist ctClass object, the corresponding target SQL statement can be configured; in the subsequent stage, the data content of the data to be monitored in the relational data is monitored according to the target SQL statement. When the operation mode in the monitored data is monitored to be the mode corresponding to the monitoring operation identifier, the operation result of the data to be monitored can be obtained.
[0056] Another preferred embodiment, this embodiment can achieve obtaining the operation result of the monitored data according to the monitoring operation identifier in the following manner: monitor the monitored data through the set data monitoring function; when the operation corresponding to the monitoring operation identifier of the monitored data is monitored, obtain the operation result.
[0057] In this embodiment, the operation result is used to indicate what operation was performed on what data in the first structure data. For example, when the monitoring operation identifier is a deletion identifier, when the data monitoring function detects that a data deletion operation has occurred on the monitored data, it is necessary to obtain which data in the monitored data has been deleted. Taking the system registration data as an example, when it is detected that a user has logged out of an account, the operation result obtained should be which user has performed the logout operation, and then in the second structure data in the subsequent steps, the same user account needs to be logged out, etc.
[0058] S130: Determine the data to be synchronized corresponding to the data to be monitored from the second structure data.
[0059] In the above step S120, the operation result of the data to be monitored in the first structure data is clarified, and then the data content included in the second structure data needs to be synchronously changed in the subsequent steps according to the operation result of the data to be monitored in the first structure data to ensure the data consistency between the first structure data and the second structure data.
[0060] Since in the specific implementation process, the first structure data and the second structure data are different system architectures, they correspond to different structure languages. Therefore, when determining the data to be synchronized corresponding to the data to be monitored from the second structure data, it is necessary to perform language mapping on the data to be monitored obtained from the first structure data, so as to map the first structure language corresponding to the data to be monitored into the second structure language that can be recognized by the second structure data, so as to determine the data to be synchronized from the second structure data.
[0061] The data to be synchronized indicates data that needs to be synchronized in the second structure data.
[0062] Specifically, the present application provides a preferred embodiment, in which the above step S130 can be implemented in the following manner: obtaining a first structural language corresponding to the data to be monitored; mapping the first structural language to obtain a second structural language corresponding to the data to be monitored; and determining the data to be synchronized from the second structural data according to the second structural language.
[0063] In this embodiment, taking the first structure data as relational data as an example, in relational data, it is implemented by structured query language (SQL for short), taking the second structure data as graph data as an example, in graph data, it is implemented by NEO4J language. Then, in one embodiment, the acquired data to be monitored is the corresponding SQL statement; and then the NEO4J statement corresponding to the data to be monitored can be obtained by mapping the SQL statement, so that the data to be synchronized can be determined from the second structure data according to the NEO4J statement. This embodiment makes language conversion more convenient and quick by mapping the corresponding structure data.
[0064] The following is a specific example of mapping SQL statements to NEO4J statements. When the monitoring operation flag of relational data is a new flag, the corresponding SQL statement is: INSERT INTO users(name, age) VALUES('Alice', 30). This line of SQL statement means adding a row of data with the name Alice and the age of 30. The corresponding mapping is converted into a NEO4J statement CREATE(users:Users{name:'Alice', age:30}). It can be seen that there is a mapping rule between the statements of relational data and the statements of graph data. According to this conversion algorithm, neo4j executable statements can be converted, so that a new edge data about the name Alice and the age of 30 can be automatically added to the corresponding data according to the neo4j executable statement.
[0065] S140: Synchronize the data to be synchronized according to the operation result.
[0066] In this embodiment, the monitoring operation identifier includes a data addition identifier and / or a data deletion identifier. That is, for a piece of data to be monitored, the operations of data addition and / or data deletion can be monitored simultaneously or separately, depending on the actual needs of the developer.
[0067] Then, the operation result corresponding to the monitoring operation identifier includes a data addition result and / or a data deletion result. That is, when the monitoring operation identifier is a data addition identifier, after executing the above step S120, the data addition result of the first structure data can be obtained; when the monitoring operation identifier is a data deletion identifier, after executing the above step S120, the data deletion result of the first structure data can be obtained. The current data addition result and / or data deletion result can be expressed in the form of deleting or adding xx data.
[0068] To more intuitively reflect the change of data, in this embodiment, for the result of adding data and / or deleting data, the corresponding first structure data will involve the change of operation parameters. Taking the first structure data as table data as an example, the corresponding operation parameter may be that the row or column of the table data has changed. Taking the first structure data as a graph file as an example, the corresponding operation parameter may be that the edge of the graph file has changed.
[0069] Specifically, the solution provided in this embodiment can further perform synchronization verification on the first structure data and the second structure data after synchronization processing is performed on the data to be synchronized according to the operation result. The specific method of performing data synchronization verification is as follows:
[0070] Determine the operation parameter identifier of the first structure data according to the operation result to obtain the first operation parameter identifier; determine the operation parameter identifier of the second structure data according to the synchronization processing result to obtain the second operation parameter identifier; perform data synchronization verification according to the first operation parameter identifier and the second operation parameter identifier.
[0071] In this embodiment, the first structure data is a table file, and the second structure data is a graph file. The operation result is the result of data addition. The number of rows of the original first structure data is 10. When the monitoring operation identifier is a data addition identifier, and the operation result includes the result of adding n rows of data, the first operation parameter identifier can be 10+n. After executing the data synchronization solution provided by this embodiment, when the side corresponding to the second operation parameter identifier is also 10+n, it indicates that the synchronization of the first structure data and the second structure data is completed. When the first operation parameter identifier and the second operation parameter identifier are inconsistent, it indicates that the synchronization of the first structure data and the second structure data has failed. Data synchronization verification can be performed by comparing the operation parameter identifiers corresponding to the first structure data and the second structure data. The implementation logic is relatively simple and the solution is easy to implement.
[0072] In another preferred embodiment, the monitoring operation identifier includes a data update identifier, the corresponding data row or edge has not changed, and the operation result corresponding to the data update identifier includes a data update result. Then, after the synchronization processing is performed on the synchronization data according to the operation result, the synchronization verification method of the first structure data and the second structure data is:
[0073] Determine the update time of the first structure data according to the data update result to obtain a first timestamp; determine the update time of the second structure data according to the data synchronization processing result to obtain a second timestamp; perform data synchronization verification according to the first timestamp and the second timestamp.
[0074] After updating the first structure data and obtaining the data update result, the first timestamp can be marked on the first structure data, and the first timestamp is used to indicate the time when the first structure data is updated; then for the second structure data, synchronization processing will be performed according to the data update result, and after the processing is completed, the updated second structure data will be marked with a second timestamp; then the data synchronization verification can be completed by comparing the first timestamp and the second timestamp.
[0075] Since in the process of implementing the data synchronization scheme, the first structure data is updated first, and then the second structure data is synchronously updated according to the update of the first structure data, the first timestamp and the second timestamp may not be exactly the same, and there will be a time difference. The embodiment of the present application can perform data synchronization verification by using the first timestamp and the second timestamp to determine whether the time difference corresponding to the first timestamp and the second timestamp is within the preset time. If so, it can be determined that the synchronization of the first structure data and the second structure data is completed, otherwise the synchronization of the first structure data and the second structure data fails. The current preset time can be 1 second, 2 seconds or 3 seconds, etc. The selection of the specific preset time length is not limited here, and the specific response time shall prevail.
[0076] In the data synchronization solution provided in the embodiment of the present application, by obtaining the pre-configured monitoring configuration information during the business system startup process, the monitoring of the operation results of the monitored data can be achieved, and by language mapping the monitored data obtained from the first structure data, the first structure language corresponding to the monitored data is mapped to the second structure language that can be recognized by the second structure data, thereby determining the data to be synchronized from the second structure data, and synchronizing the data to be synchronized according to the operation results. The processing operations of the two data structures are completed in one transaction, and the timeliness is at the second level. There is no problem of scheduled task cycle, which improves the data synchronization efficiency. In addition, the present application only needs to dynamically call the data synchronization plug-in when starting the business system to complete the injection of the agent, which simplifies the deployment structure and saves deployment resources.
[0077] The data synchronization method provided in this embodiment configures the monitoring configuration information locally in advance, and can obtain the monitoring data identifier and monitoring operation identifier contained in the monitoring configuration information, thereby determining the data to be monitored from the first structure data through the monitoring data identifier, and then obtaining the operation result of the data to be monitored through the monitoring operation identifier, and then determining the data to be synchronized corresponding to the data to be monitored from the second structure data, and finally synchronizing the data to be synchronized according to the operation result, so as to achieve the purpose of keeping the data content in the first structure data and the second structure data consistent. The solution provided in this embodiment does not need to deploy complex processing tools in the first structure data, and can achieve the synchronization of the first structure data and the second structure data through the monitoring configuration information, which has the beneficial effect of improving the efficiency of data synchronization and simplifying the deployment structure of the solution.
[0078] Figure 2 1 is a schematic diagram of a data synchronization device provided in an embodiment of the present application, and the device is suitable for executing the data synchronization method provided in an embodiment of the present application. Figure 2 As shown, the device may specifically include: an information acquisition module 210, a result acquisition module 220, a data determination module 230 and a synchronization processing module 240, wherein:
[0079] The information acquisition module 210 is used to acquire monitoring configuration information, wherein the monitoring configuration information includes a monitoring data identifier and a monitoring operation identifier;
[0080] A result acquisition module 220, configured to determine the data to be monitored from the first structure data according to the monitoring data identifier, and to acquire the operation result of the data to be monitored according to the monitoring operation identifier;
[0081] A data determination module 230, configured to determine the data to be synchronized corresponding to the data to be monitored from the second structure data;
[0082] The synchronization processing module 240 is used to perform synchronization processing on the data to be synchronized according to the operation result.
[0083] The data synchronization device provided in this embodiment configures the monitoring configuration information locally in advance, and can obtain the monitoring data identifier and the monitoring operation identifier contained in the monitoring configuration information, thereby determining the data to be monitored from the first structure data through the monitoring data identifier, and then obtaining the operation result of the data to be monitored through the monitoring operation identifier, and then determining the data to be synchronized corresponding to the data to be monitored from the second structure data, and finally synchronizing the data to be synchronized according to the operation result, so as to achieve the purpose of keeping the data content in the first structure data and the second structure data consistent. The solution provided in this embodiment does not need to deploy complex processing tools in the first structure data, and can achieve the synchronization of the first structure data and the second structure data through the monitoring configuration information, which has the beneficial effect of improving the efficiency of data synchronization and simplifying the deployment structure of the solution.
[0084] In one embodiment, the result acquisition module 220 includes an information acquisition unit and a data determination unit, wherein:
[0085] An information acquisition unit, used to acquire at least one class name information in the first structure data;
[0086] A data determination unit is used to determine the data to be monitored according to the monitoring data identifier and at least one of the category name information.
[0087] In one embodiment, the data determination unit is specifically used to obtain, for at least one of the class name information, a current compilation identifier obtained after performing a compilation operation on the current class name information; when the current compilation identifier and the monitoring data identifier are consistent, determining the data to be monitored based on the current compilation identifier; when the current compilation identifier and the monitoring data identifier are inconsistent, performing an operation of compiling the next class name information.
[0088] In one embodiment, the result acquisition module 220 further includes a data monitoring unit and a result acquisition unit, wherein:
[0089] A data monitoring unit, used for monitoring the data to be monitored through a set data monitoring function;
[0090] The result acquisition unit is used to acquire the operation result when the operation corresponding to the monitoring operation identifier is monitored and performed on the data to be monitored.
[0091] In one embodiment, the data determination module 230 includes a language acquisition unit and a data determination unit, wherein:
[0092] A language acquisition unit, used for acquiring a first structured language corresponding to the data to be monitored;
[0093] The language acquisition unit is further used to perform mapping processing on the first structure language to obtain a second structure language corresponding to the data to be monitored;
[0094] A data determining unit is used to determine the data to be synchronized from the second structure data according to the second structure language.
[0095] In one embodiment, the monitoring operation identifier includes a data addition identifier and / or a data deletion identifier, and correspondingly, the operation result includes a data addition result and / or a data deletion result; the device further includes: an identifier acquisition module and a data verification module, wherein:
[0096] an identification acquisition module, used to determine the operation parameter identification of the first structure data according to the operation result, and obtain a first operation parameter identification;
[0097] The identification acquisition module is further used to determine the operation parameter identification of the second structure data according to the synchronization processing result, and obtain the second operation parameter identification;
[0098] A data verification module is used to perform data synchronization verification according to the first operation parameter identifier and the second operation parameter identifier.
[0099] In one embodiment, the monitoring operation identifier includes a data update identifier, and correspondingly, the operation result includes a data update result; the device further includes a timestamp acquisition module, wherein:
[0100] A timestamp acquisition module, used to determine the update time of the first structure data according to the data update result, and obtain a first timestamp;
[0101] The timestamp acquisition module is further used to determine the update time of the second structure data according to the data synchronization processing result to obtain a second timestamp;
[0102] The data verification module is further used to perform data synchronization verification according to the first timestamp and the second timestamp.
[0103] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example for illustration. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. The specific working process of the functional modules described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.
[0104] An embodiment of the present application also provides an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the data synchronization method described in any embodiment of the present application.
[0105] An embodiment of the present application further provides a computer-readable medium, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the data synchronization method described in any embodiment of the present application when executed.
[0106] Reference below Figure 3 , Figure 3 1 is a schematic diagram of the structure of an electronic device of a data synchronization method provided in an embodiment of the present application, and shows a schematic diagram of the structure of a computer system 500 suitable for implementing the electronic device of the embodiment of the present application. Figure 3 The electronic device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.
[0107] like Figure 3 As shown, the computer system 500 includes a central processing unit (CPU) 501, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 502 or a program loaded from a storage part 508 into a random access memory (RAM) 503. In the RAM 503, various programs and data required for the operation of the system 500 are also stored. The CPU 501, the ROM 502, and the RAM 503 are connected to each other via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.
[0108] The following components are connected to the I / O interface 505: an input section 506 including a keyboard, a mouse, etc.; an output section 507 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 508 including a hard disk, etc.; and a communication section 509 including a network interface card such as a LAN card, a modem, etc. The communication section 509 performs communication processing via a network such as the Internet. A drive 510 is also connected to the I / O interface 505 as needed. A removable medium 511, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 510 as needed, so that a computer program read therefrom is installed into the storage section 508 as needed.
[0109] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 509, and / or installed from the removable medium 511. When the computer program is executed by the central processing unit (CPU) 501, the above-mentioned functions defined in the system of the present application are executed.
[0110] It should be noted that the computer-readable medium shown in the present application may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present application, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, etc., or any suitable combination of the above.
[0111] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the above-mentioned module, program segment or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.
[0112] The modules and / or units involved in the embodiments described in the present application may be implemented by software or hardware. The modules and / or units described may also be arranged in a processor, for example, it may be described as follows: a processor includes an information acquisition module, a result acquisition module, a data determination module, and a synchronization processing module. The names of these modules do not, in some cases, constitute limitations on the modules themselves.
[0113] As another aspect, the present application also provides a computer-readable medium, which may be included in the device described in the above embodiment; or it may exist independently without being assembled into the device. The above computer-readable medium carries one or more programs, and when the above one or more programs are executed by a device, the device includes: obtaining monitoring configuration information, the monitoring configuration information includes a monitoring data identifier and a monitoring operation identifier; determining the data to be monitored from the first structure data according to the monitoring data identifier, and obtaining the operation result of the data to be monitored according to the monitoring operation identifier; determining the data to be synchronized corresponding to the data to be monitored from the second structure data; and synchronizing the data to be synchronized according to the operation result.
[0114] According to the technical solution of this embodiment, by configuring the monitoring configuration information locally in advance, the monitoring data identifier and the monitoring operation identifier contained in the monitoring configuration information can be obtained, so that the data to be monitored can be determined from the first structure data through the monitoring data identifier, and then the operation result of the data to be monitored can be obtained through the monitoring operation identifier, and then the data to be synchronized corresponding to the data to be monitored can be determined from the second structure data, and finally the data to be synchronized can be synchronized according to the operation result, so as to achieve the purpose of keeping the data content in the first structure data and the second structure data consistent. The solution provided by this embodiment does not need to deploy complex processing tools in the first structure data, and the synchronization of the first structure data and the second structure data can be achieved through the monitoring configuration information, which has the beneficial effect of improving the efficiency of data synchronization and simplifying the deployment structure of the solution.
[0115] The above specific implementations do not constitute a limitation on the protection scope of this application. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may occur depending on design requirements and other factors. Any modifications, equivalent substitutions and improvements made within the spirit and principles of this application should be included in the protection scope of this application.
Claims
1. A data synchronization method, characterized in that: include: Acquire monitoring configuration information, wherein the monitoring configuration information includes a monitoring data identifier and a monitoring operation identifier; Determine the data to be monitored from the first structure data according to the monitoring data identifier, and obtain the operation result of the data to be monitored according to the monitoring operation identifier; Determine the data to be synchronized corresponding to the data to be monitored from the second structure data; The data to be synchronized is synchronized according to the operation result.
2. The data synchronization method according to claim 1, characterized in that: The step of determining the data to be monitored from the first structure data according to the monitoring data identifier includes: Obtain at least one class name information in the first structure data; The data to be monitored is determined according to the monitoring data identifier and at least one of the class name information.
3. The data synchronization method according to claim 2, characterized in that: The determining the data to be monitored according to the monitoring data identifier and at least one of the class name information includes: For at least one piece of current class name information, obtaining a current compilation identifier obtained after a compilation operation is performed on the current class name information; When the current compilation identifier and the monitoring data identifier are consistent, determining the data to be monitored according to the current compilation identifier; When the current compilation identifier and the monitoring data identifier are inconsistent, an operation of compiling the next category name information is performed.
4. The data synchronization method according to claim 1, characterized in that: The obtaining the operation result of the to-be-monitored data according to the monitoring operation identifier includes: Monitoring the data to be monitored by using a set data monitoring function; When the operation corresponding to the monitoring operation identifier is monitored and performed on the data to be monitored, the operation result is obtained.
5. The data synchronization method according to claim 1, characterized in that: The step of determining the data to be synchronized corresponding to the data to be monitored from the second structure data includes: Acquire a first structured language corresponding to the data to be monitored; Performing mapping processing on the first structure language to obtain a second structure language corresponding to the data to be monitored; The data to be synchronized is determined from the second structured data according to the second structured language.
6. The data synchronization method according to claim 1, characterized in that: The monitoring operation identifier includes a data addition identifier and / or a data deletion identifier, and correspondingly, the operation result includes a data addition result and / or a data deletion result; After synchronizing the data to be synchronized according to the operation result, the method further includes: Determine the operation parameter identifier of the first structure data according to the operation result to obtain a first operation parameter identifier; Determine the operation parameter identifier of the second structure data according to the synchronization processing result, and obtain a second operation parameter identifier; A data synchronization check is performed according to the first operation parameter identifier and the second operation parameter identifier.
7. The data synchronization method according to claim 1, characterized in that: The monitoring operation identifier includes a data update identifier, and correspondingly, the operation result includes a data update result; After synchronizing the data to be synchronized according to the operation result, the method further includes: Determine the update time of the first structure data according to the data update result to obtain a first timestamp; Determine the update time of the second structure data according to the data synchronization processing result, and obtain a second timestamp; A data synchronization check is performed according to the first timestamp and the second timestamp.
8. A data synchronization device, characterized in that: include: An information acquisition module, used to acquire monitoring configuration information, wherein the monitoring configuration information includes a monitoring data identifier and a monitoring operation identifier; A result acquisition module, used to determine the data to be monitored from the first structure data according to the monitoring data identifier, and to acquire the operation result of the data to be monitored according to the monitoring operation identifier; A data determination module, used to determine the data to be synchronized corresponding to the data to be monitored from the second structure data; A synchronization processing module is used to perform synchronization processing on the data to be synchronized according to the operation result.
9. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the data synchronization method according to any one of claims 1 to 7.
10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the data synchronization method as described in any one of claims 1 to 7 is implemented.
Citation Information
Patent Citations
Heterogeneous database synchronization method and device, computer equipment and storage medium
CN111382201A
Data synchronization method and device, electronic equipment and storage medium
CN113407637A
Data synchronization method and device, electronic equipment and computer readable storage medium
CN116455917A
Database synchronous operation method, device and equipment and computer medium
CN117235177A
Data synchronization method and apparatus
US20180210937A1
