Method and device for synchronizing data

Abstract

The invention discloses a data synchronization method and device, and relates to the technical field of big data. A specific embodiment of the method comprises the steps of filtering out target log information matched with a preset filtering strategy from data logs collected by a plurality of first data sources, and obtaining the target log information through a preset subscription service; and analyzing the change information from the target log information, and synchronizing the to-be-synchronized data corresponding to the change information to the second data source. The original data logs are correspondingly filtered, and the filtered logs and the synchronization data are decoupled through the subscription service, so that the data synchronization efficiency is improved, and resources consumed by the synchronization data are reduced.

Description

Technical Field

[0001] The present invention relates to the field of big data technology, and in particular to a method and device for synchronizing data. Background Art

[0002] In data processing scenarios of Internet applications, it is usually necessary to aggregate and synchronize data from multiple different data sources to a target data source. For example, data from data sources located in different geographical locations need to be synchronized to a target data source so that all data can be managed uniformly through the target data source.

[0003] Currently, the method for synchronizing data from different data sources to a target data source mainly utilizes a data processing interface to synchronize data to the target data source. When the amount of data to be processed is large and the network performance in different geographical locations is different, the existing method for synchronizing data has the problems of low efficiency in synchronizing data and waste of network resources. Summary of the invention

[0004] In view of this, an embodiment of the present invention provides a method and device for synchronizing data, which can filter out target log information matching a preset filtering strategy from data logs collected by multiple first data sources, obtain the target log information through a preset subscription service, parse the change information from the target log information, and synchronize the data to be synchronized corresponding to the change information to a second data source. By filtering the original data logs accordingly and decoupling the processing of the filtered logs from the synchronized data through the subscription service, the efficiency of synchronized data is improved and the resources consumed by synchronized data are reduced.

[0005] To achieve the above-mentioned purpose, according to one aspect of an embodiment of the present invention, a method for synchronizing data is provided, characterized in that it includes: collecting data logs from multiple first data sources, filtering out target log information matching a preset filtering policy from the data logs, and storing the target log information; obtaining the stored target log information that has changed through a preset subscription service; parsing the change information from the target log information that has changed, and synchronizing the data to be synchronized corresponding to the change information to a second data source.

[0006] Optionally, the method for synchronizing data also includes: providing at least one log processing layer and at least one synchronization business layer that are decoupled from each other, wherein each of the log processing layers is configured with a preset filtering policy corresponding to the synchronization business layer, and each of the synchronization business layers is bound to at least one of the subscription services; executing the step of filtering out target log information that matches the preset filtering policy from the data log through any idle log processing layer; and the synchronization business layer executes the step of obtaining the stored target log information that has changed and parsing the change information from the target log information that has changed.

[0007] Optionally, the preset filtering strategy includes data identification filtering, data processing instruction filtering, regular expression filtering, multiple filtering types in custom filtering strategy and the execution order of the multiple filtering types; filtering out target log information matching the preset filtering strategy from the data log includes: parsing out the multiple filtering types included in the preset filtering strategy and the execution order of the multiple filtering types; filtering out target log information matching each of the parsed filtering types from the data log according to the execution order of the multiple filtering types.

[0008] Optionally, the method for synchronizing data further includes: building an index for the log file corresponding to the collected data log, generating an index file for the log file; in response to searching for the log file of the data log or the target log information, locating the log file through the index file.

[0009] Optionally, the method for synchronizing data further includes: searching for log files corresponding to the data log or the target log information that meet the set cleaning policy; if found, cleaning the log files that meet the set cleaning policy; and updating the index file.

[0010] Optionally, the method for synchronizing data further includes: obtaining a synchronization result of the second data source for the data to be synchronized; when the synchronization result indicates a synchronization failure, locating a log file associated with the failure information in the index file according to the failure information corresponding to the synchronization failure indicated by the synchronization result; and re-filtering target log information that matches the preset filtering policy from the log file associated with the failure information.

[0011] Optionally, the method for synchronizing data further comprises:

[0012] When the target log information is written into a new data log file, the new log file is compressed; and the acquiring of the stored target log information with changes includes: acquiring the target log information with changes from the compressed new log file.

[0013] To achieve the above object, according to a second aspect of an embodiment of the present invention, a device for synchronizing data is provided, characterized in that it includes: a log processing module and a data synchronization module; wherein,

[0014] The log processing module is used to collect data logs from multiple first data sources, filter out target log information matching a preset filtering policy from the data logs, and store the target log information;

[0015] The synchronization data module is used to obtain the stored target log information with changes through a preset subscription service; parse the change information from the target log information with changes, and synchronize the data to be synchronized corresponding to the change information to the second data source.

[0016] Optionally, the device for synchronizing data is also used to set up at least one log processing layer and at least one synchronization business layer that are decoupled from each other, wherein each of the log processing layers is configured with a preset filtering policy corresponding to the synchronization business layer, and each of the synchronization business layers is bound to at least one of the subscription services; the step of filtering out target log information matching the preset filtering policy from the data log is executed through any idle log processing layer; the synchronization business layer executes the steps of obtaining the stored target log information that has changed and parsing the change information from the target log information that has changed.

[0017] Optionally, the device for synchronizing data includes the preset filtering strategy including data identification filtering, data processing instruction filtering, regular expression filtering, multiple filtering types in the custom filtering strategy and the execution order of the multiple filtering types; filtering out the target log information matching the preset filtering strategy from the data log includes: parsing out the multiple filtering types included in the preset filtering strategy and the execution order of the multiple filtering types; filtering out the target log information matching each of the parsed filtering types from the data log according to the execution order of the multiple filtering types.

[0018] Optionally, the device for synchronizing data is further used to build an index for the log file corresponding to the collected data log, and generate an index file for the log file; in response to searching for the log file of the data log or the target log information, locate the log file through the index file.

[0019] Optionally, the device for synchronizing data is further used to search for log files corresponding to the data log or the target log information that meet the set cleaning policy; if found, clean up the log files that meet the set cleaning policy; and update the index file.

[0020] Optionally, the device for synchronizing data is also used to obtain a synchronization result of the second data source for the data to be synchronized; when the synchronization result indicates a synchronization failure, based on the failure information corresponding to the synchronization failure indicated by the synchronization result, locate the log file associated with the failure information according to the index file; and re-filter the target log information that matches the preset filtering policy from the log file associated with the failure information.

[0021] Optionally, the device for synchronizing data is also used to compress the new log file when the target log information is written into a new data log file; the obtaining of the stored target log information that has changed includes: obtaining the target log information that has changed from the compressed new log file.

[0022] To achieve the above-mentioned purpose, according to the third aspect of an embodiment of the present invention, there is provided an electronic device for synchronizing data, characterized in that it comprises: one or more processors; a storage device for storing one or more programs, when the one or more programs are executed by the one or more processors, the one or more processors implement any of the methods described in the above-mentioned methods for synchronizing data.

[0023] To achieve the above-mentioned purpose, according to a fourth aspect of an embodiment of the present invention, a computer-readable medium is provided, on which a computer program is stored, characterized in that when the program is executed by a processor, any method described in the above-mentioned method for synchronizing data is implemented.

[0024] An embodiment of the above invention has the following advantages or beneficial effects: it is able to filter out target log information that matches a preset filtering strategy from data logs collected from multiple first data sources, and obtain the target log information through a preset subscription service; parse the change information from the target log information, and synchronize the data to be synchronized corresponding to the change information to a second data source. By filtering the original data logs accordingly and decoupling the processing of the filtered logs from the synchronized data through the subscription service, the efficiency of synchronized data is improved and the resources consumed by synchronized data are reduced.

[0025] The further effects of the above-mentioned non-conventional optional manner will be described below in conjunction with the specific implementation manner. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] The accompanying drawings are used to better understand the present invention and do not constitute an improper limitation of the present invention.

[0027] Figure 1 It is a flowchart of a method for synchronizing data provided by an embodiment of the present invention;

[0028] Figure 2 It is a schematic diagram of a data synchronization process provided by an embodiment of the present invention;

[0029] Figure 3 is a schematic diagram of another process of synchronizing data provided by an embodiment of the present invention;

[0030] Figure 4 is a schematic diagram of the structure of a device for synchronizing data provided by an embodiment of the present invention;

[0031] Figure 5 is an exemplary system architecture diagram to which embodiments of the present invention may be applied;

[0032] Figure 6 It is a schematic diagram of the structure of a computer system of a terminal device or a server suitable for implementing an embodiment of the present invention. DETAILED DESCRIPTION

[0033] The following is a description of exemplary embodiments of the present invention in conjunction with the accompanying drawings, including various details of the embodiments of the present invention to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present invention. Similarly, for clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0034] It should be noted that the collection, collection, updating, analysis, processing, use, transmission, storage and other aspects of user personal information involved in the technical solution of the present invention are in compliance with the provisions of relevant laws and regulations, are used for legitimate purposes, and do not violate public order and good morals. Necessary measures are taken for user personal information to prevent illegal access to user personal information data and maintain the security of user personal information and network security.

[0035] It should be noted that in the technical solution of the present invention, the collection, use, storage, sharing and transfer of user personal information involved are in compliance with the provisions of relevant laws and regulations, and it is necessary to inform the user and obtain the user's consent or authorization. When applicable, the user's personal information is de-identified and / or anonymized and / or encrypted.

[0036] In order to count the usage of our products / services, we will aggregate, analyze and use the technically processed user data and share the processed statistical information with third parties. We will use secure encryption technology and other methods to ensure that the recipient of the information cannot re-identify a specific individual.

[0037] like Figure 1 As shown, an embodiment of the present invention provides a method for synchronizing data, which may include the following steps:

[0038] Step S101: collecting data logs from a plurality of first data sources, filtering out target log information matching a preset filtering strategy from the data logs, and storing the target log information.

[0039] Specifically, in a system of multiple application scenarios, it is necessary to synchronize data from different sub-data sources (i.e., the first data source) to a total data source (i.e., the second data source). Different sub-data sources may be located in different geographical locations. Taking a logistics system as an example, the total logistics data managed by the logistics system needs to be collected / synchronized from warehouse data at multiple different addresses. Due to the large differences in network performance in different geographical locations, the data processing interface of the existing method is used to synchronize data to the target data source. When a large amount of data is processed, this results in low efficiency in synchronizing data, high synchronization failure rate, and waste of network resources.

[0040] The embodiment of the present invention utilizes a method of decoupling processing data logs and synchronizing data, thereby overcoming the problems of low efficiency of synchronizing data, high synchronization failure rate, and waste of network resources in the existing method.

[0041] Specifically, data logs are collected from multiple first data sources, and the data logs are used to record operations on the data sources. Each data source may have one or more data logs of its own. For example, the data log of data source A contains changes to one or more data tables of data source A in data source A, as well as changes to one or more data items. Specifically, the data log may include the operation timestamp of the data source, the server ID associated with the data source, the SQL (Structured Query Language, referred to as SQL) command executed to operate the data source, the operation duration of the data source, etc. That is, the data change (addition, deletion, modification, etc.) of the first data source can be parsed from the data log of the first data source, so that the second data source performs data synchronization on the first data source according to the data change information contained in the data log. For example: the log information contained in the data log of the first data source A includes 1000 log data indicating that 1000 new data items are added to the data table A of the data source A, and each log data item indicates the specific data content of each new data item; then, according to the specific data content of each new data item in the 1000 log data items, the 1000 new data items can be synchronized (i.e. added) to the data table A corresponding to the data source A, thereby completing data synchronization. The data log format can be text, binary, etc. The present invention does not limit the specific data content and format of the data log; it can be understood that in actual application scenarios, the data tables processed are of a large order of magnitude, and the data to be synchronized often have correlations between multiple data tables; therefore, the method of obtaining the data to be synchronized from the data log reduces the complexity of obtaining the data to be synchronized and improves the efficiency of synchronizing the data.

[0042] Furthermore, the method of collecting data logs corresponding to the first data sources from multiple first data sources can be collected by downloading according to a set time period, or by triggering data push through conditions such as data changes, etc.

[0043] Furthermore, after obtaining the data log, target log information matching the preset filtering policy is filtered out from the data log, and the target log information is stored; in an embodiment of the present invention, at least one log processing layer and at least one synchronization business layer that are decoupled from each other are provided, and the log processing layer is used to perform the steps of filtering out target log information matching the preset filtering policy on the data log, and storing the target log information, wherein the preset filtering policy is one or more filtering policies configured by the synchronization business layer.

[0044] It can be understood that the preset filtering strategy is to execute data processing instructions for data changes (such as SQL instructions and specific data operated by instructions, etc.) for data identifiers containing data to be synchronized (for example, data sources, data tables, fields, field values, etc.), and use regular expressions as filtering conditions, wherein regular expression filtering, for example, filters data from multiple tables, or further filters out information that meets a certain set condition from the change data contained in the log, or sets a custom filtering strategy based on the business scenario, etc.; further, the filtering execution order for multiple filtering types can be set, for example: first execute data identifier filtering, then regular expression filtering, and finally execute data processing instruction filtering, etc.; it can be understood that the execution order is directly related to the business type of the specific application scenario, and the present invention does not limit the specific types of multiple filtering types and the specific execution order. That is, the preset filtering strategy includes multiple filtering types in data identification filtering, data processing instruction filtering, regular expression filtering, and custom filtering strategy, and the execution order of the multiple filtering types; filtering out target log information matching the preset filtering strategy from the data log includes: parsing out the multiple filtering types included in the preset filtering strategy and the execution order of the multiple filtering types; filtering out target log information matching each of the parsed filtering types from the data log according to the execution order of the multiple filtering types. Filtering the data log by multiple filtering types and the execution order of the multiple filtering types improves the efficiency of filtering the log.

[0045] Furthermore, a configuration platform or page can be provided to the management end of the synchronization business layer, so that the management end of the business synchronization layer can configure one or more corresponding preset filtering strategies and execution order and other information based on the configuration platform or page, and send it to the log processing layer, so that the log processing layer can filter out the target log information that matches each of the parsed filtering types from the data log based on one or more preset filtering strategies and filtering type execution order.

[0046] Since there is a large amount of log information not associated with the data changes of specific businesses in various operations on data sources recorded in data logs, for example, the data source contains 1000 data tables, and 10 data tables are directly related to the synchronization data of business A, then the log information of data changes of other data tables other than these 10 data tables is log information not associated with the data changes of business A, or the data log also contains other data operation information not associated with the synchronization data: data query, data reading, etc. Therefore, by performing filtering operations on the data logs that match the preset filtering strategy, compared with directly obtaining the data logs and parsing the log files one by one, the data volume of the logs is reduced to a large extent, thereby reducing the network resource consumption of transmitting the logs, and reducing the calculation amount of processing data at the synchronization business layer; and the log processing layer (such as a log processing process or a log processing module, etc.) that processes the logs is decoupled from the synchronization business layer (such as a synchronization business process or a synchronization business module, etc.) of the synchronization data, thereby improving the flexibility of changing the filtering rules, and when the amount of data to be synchronized is large within a short time range, the efficiency of synchronizing the data is improved to a large extent.

[0047] Furthermore, after filtering out target log information matching a preset filtering policy from the data log, the target log information is stored; wherein the storage method may be stored in a cache, a storage device, etc., and the target log information is usually stored in a format such as a log file or a log data source.

[0048] Step S102: Obtain stored target log information with changes through a preset subscription service.

[0049] Specifically, the log processing layer may be used to provide a preset subscription service to notify the synchronization service layer to perform data synchronization based on the filtered target log information.

[0050] In one embodiment of the present invention, the preset subscription service can be a subscription based on the CDC (Change Data Capture) method of the data source, and when the data of the data source is determined to have changed through the subscription service, the target log information corresponding to the data change is obtained; or, in another embodiment of the present invention, the predetermined subscription service can also be a service for monitoring the changes of the log file of the target log information, and when the log file is monitored to have changes (file modification time change, file size change, etc.), the synchronization business layer is notified to obtain the log file, that is, to obtain the stored target log information with changes. When the target log information is written into a new data log file, the data log file can be provided to the synchronization business layer through the subscription service.

[0051] Step S103: parsing the change information from the target log information with changes, and synchronizing the to-be-synchronized data corresponding to the change information to the second data source.

[0052] Specifically, the target log information obtained after filtering contains specific data change information, such as: deleting multiple data in data table B of data source A and the specific data of each deleted data; or multiple data in data table C of data source A are changed, the data before the change and the data after the change; or multiple data are added to data table D of data source A and the specific content of each newly added data, etc. It can be understood that in actual application scenarios, multiple data sources or multiple data tables of the same data source can have an association relationship, and data changes of multiple data sources or multiple data tables of the same data source can also be obtained through the change information in the target log information.

[0053] Furthermore, based on the specific data (i.e., data to be synchronized) contained in the obtained change information, the data to be synchronized corresponding to the change information is synchronized to the second data source, so that the latest data after synchronization of each first data source can be obtained through the second data source, so that subsequent further statistics, analysis, use and other data operations can be performed based on the latest data.

[0054] Furthermore, in the process of synchronizing the to-be-synchronized data corresponding to the change information to the second data source, the synchronization can be completed by obtaining the set synchronization strategy, which includes, for example, a mapping relationship between the field values ​​of the data table in the first data source and the field values ​​of the data table in the second data source, or a conversion relationship between the format of the field values ​​of the data table in the first data source and the format of the field values ​​of the data table in the second data source, etc. Among them, the second data source can be a data source corresponding to various types of data components, such as: message queue MQ (Message Queue), relational database MYSQL, data search and analysis engine ES (Elasticsearch), data analysis system HIVE, custom data source, etc. The flexibility and scalability of synchronized data are further improved.

[0055] like Figure 2 As shown, an embodiment of the present invention provides a process for synchronizing data, which may include the following steps:

[0056] For each of the one or more log processing layers (log processing layers 1 to N), steps S201 to S204 are executed:

[0057] Step S201: collecting data logs from multiple first data sources.

[0058] Step S202: determine whether the log needs to be filtered, and if so, execute step S203; otherwise, directly execute the data synchronization operation based on the log file.

[0059] Preferably, it is possible to determine whether a data log of a first data source needs to be filtered based on the value of the filter switch option indicated by the preset filter policy (for example: the value of on is true, the value of off is false), or based on information such as the data source identifier; if it is determined that filtering is required, the target log information matching the preset filter policy is filtered out from the data log; otherwise, subsequent data synchronization operations are performed directly based on the original data log. By determining whether filtering is required, the flexibility, configurability and personalization of managing data synchronization are improved.

[0060] Step S203: filtering out target log information matching a preset filtering strategy from the data log.

[0061] Step S204: storing the target log information.

[0062] Specifically, the description of collecting data logs and filtering log files in steps S201 to S204 is consistent with the description of step S101 and will not be repeated here.

[0063] For each of the one or more synchronization service layers (synchronization service layers 1 to N), steps S205 to S207 are executed:

[0064] Step S205: Obtain stored target log information with changes through a preset subscription service.

[0065] Step S206: parsing the changed target log information to obtain the changed target log information.

[0066] Step S207: Synchronize the to-be-synchronized data corresponding to the change information to the second data source.

[0067] Specifically, the description of synchronizing data based on the change information in the target log information in steps S205 to S207 is consistent with the description of steps S102 to S103 , and will not be repeated here.

[0068] The description of step S201 to step S207 is: at least one log processing layer and at least one synchronization business layer are decoupled from each other, wherein each of the log processing layers is configured with a preset filtering strategy corresponding to the synchronization business layer, and each of the synchronization business layers is bound to at least one of the subscription services; the step of filtering out target log information matching the preset filtering strategy from the data log is executed by any idle log processing layer; the synchronization business layer executes the steps of acquiring the stored target log information with changes and parsing the change information from the target log information with changes. It can be understood that steps S201 to step S204 and steps S205 to step S207 are executed by the log processing layer and the synchronization business layer respectively, and multiple synchronization business layers can concurrently execute the steps of synchronizing data, so the sequence numbers of the above steps are only examples.

[0069] like Figure 3 As shown, an embodiment of the present invention provides another method for synchronizing data, which may include the following steps:

[0070] Step S301: collecting data logs from a plurality of first data sources, filtering out target log information matching a preset filtering strategy from the data logs, and storing the target log information.

[0071] Specifically, the description of collecting data logs from multiple first data sources, filtering out target log information matching a preset filtering policy from the data logs, and storing the target log information is consistent with the description of step S101 and is not limited here.

[0072] Step S302: construct an index for the log file corresponding to the collected data log, and generate an index file for the log file.

[0073] Specifically, after collecting one or more data logs corresponding to multiple first data sources, when the data logs are stored in log files, preferably, an index is constructed for the log files corresponding to the logs, and an index file for the log files is generated based on the index. By constructing the index and the index file, the efficiency of locating one or more log files is improved when the order of magnitude of the log files is large. After locating the log files, the log information contained in the log files can be further processed (for example, filtering or reading information, etc.); preferably, the target log information obtained by filtering can also be stored in a new log file, and an index is constructed for the new log file and added to the index file; wherein each index file can only store the unique identifier of each log file. That is, an index is constructed for the log files corresponding to the collected data logs, and an index file for the log files is generated; in response to searching for the log files of the data logs or the target log information, the log files are located through the index files.

[0074] Further preferably, for log files, a log file corresponding to the data log or the target log information that satisfies a set cleaning strategy is searched; if found, the log file that satisfies the set cleaning strategy is cleaned; and the index file is updated.

[0075] Among them, a cleanup strategy is set, for example, the current time of the creation time distance includes log files that exceed a set time range (for example, one month, three months, etc.), or log files that are marked as invalid, or log files whose file size exceeds a set threshold, etc.; that is, the log files corresponding to the data log or the target log information that meet the set cleanup strategy are searched; if found, the log files that meet the set cleanup strategy are cleaned, and the cleaning includes deleting them from the storage medium (cache or hard disk), thereby reducing the consumption of storage resources.

[0076] Further, preferably, when the target log information is written into a new data log file, the new log file is compressed; the acquiring of the stored target log information with changes includes: acquiring the target log information with changes from the compressed new log file. After parsing the log file and filtering, the new data log file can be compressed, and the compression method, for example, adopts a compression method based on Snappy. Through compression processing, the network resources consumed by transmitting the new data log file are further reduced.

[0077] Step S303: acquiring stored target log information with changes through a preset subscription service, parsing change information from the target log information with changes, and synchronizing the data to be synchronized corresponding to the change information to the second data source.

[0078] Specifically, the specific description of synchronizing the to-be-synchronized data corresponding to the change information to the second data source is consistent with the description of step S103 and will not be repeated here.

[0079] Step S304: Obtain the synchronization result of the second data source for the data to be synchronized.

[0080] Specifically, after executing data synchronization, the synchronization result of the second data source for the data to be synchronized is obtained; when performing data synchronization on the second data source at the synchronization business layer, operations such as writing to the data source may result in synchronization failure due to the performance and environment of the physical device to which the data source belongs, that is, the synchronization result for the data to be synchronized may indicate synchronization success or synchronization failure.

[0081] Step S305: Determine whether the synchronization result indicates that the synchronization has failed. If so, execute step S306; otherwise, the data synchronization is completed.

[0082] Step S306: According to the failure information corresponding to the synchronization failure indicated by the synchronization result, locate the log file associated with the failure information in the index file, and re-execute step S301 and subsequent steps of synchronizing data.

[0083] Specifically, when it is determined that the synchronization result indicates a synchronization failure, contextual information of the synchronization failure can be obtained, such as: log files associated with the synchronization failure, data that has been synchronized, data synchronized when the synchronization fails, and other information; thereby, the log file associated with the failure information can be located according to the index file, and the step of filtering the log associated with step S301 can be re-executed, or a new log file associated with the failure information can be located after filtering according to the index file, and the part corresponding to the failure information can be parsed from the new log file, and the step of synchronizing the data can be re-executed.

[0084] That is, obtaining the synchronization result of the second data source for the data to be synchronized; when the synchronization result indicates a synchronization failure, locating the log file associated with the failure information in the index file according to the failure information corresponding to the synchronization failure indicated by the synchronization result; and re-filtering the target log information that matches the preset filtering policy from the log file associated with the failure information.

[0085] From the description of steps S301 to S306, it can be seen that the embodiments of the present invention can automatically determine the synchronization results of data synchronization, and in the event of data synchronization failure, automatically locate the log file associated with the failure information according to the constructed index file, thereby automatically re-executing the data synchronization process, thereby greatly improving the fault tolerance and automation of synchronized data, thereby improving the efficiency of synchronized data.

[0086] like Figure 4 As shown, an embodiment of the present invention provides a device 400 for synchronizing data, including: a log processing module 401 and a data synchronization module 402; wherein,

[0087] The log processing module 401 is used to collect data logs from multiple first data sources, filter out target log information matching a preset filtering policy from the data logs, and store the target log information;

[0088] The data synchronization module 402 is used to obtain the stored target log information with changes through a preset subscription service; parse the change information from the target log information with changes, and synchronize the data to be synchronized corresponding to the change information to the second data source.

[0089] An embodiment of the present invention also provides an electronic device for synchronizing data, comprising: one or more processors; a storage device for storing one or more programs, when the one or more programs are executed by the one or more processors, the one or more processors implement the method provided in any of the above embodiments.

[0090] An embodiment of the present invention further provides a computer-readable medium on which a computer program is stored. When the program is executed by a processor, the method provided in any of the above embodiments is implemented.

[0091] Figure 5 An exemplary system architecture 500 is shown to which the method or apparatus for synchronizing data according to the embodiments of the present invention may be applied.

[0092] like Figure 5 As shown, system architecture 500 may include terminal devices 501, 502, 503, a network 504 and a server 505. Network 504 is used to provide a medium for communication links between terminal devices 501, 502, 503 and server 505. Network 504 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0093] Users can use terminal devices 501, 502, 503 to interact with server 505 through network 504 to receive or send messages, etc. Various client applications can be installed on terminal devices 501, 502, 503, such as e-commerce client applications, web browser applications, search applications, instant messaging tools, and email clients.

[0094] The terminal devices 501 , 502 , and 503 may be various electronic devices having a display screen and supporting various client applications, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, and the like.

[0095] The server 505 may be a server that provides various services, such as a background management server that supports the client applications used by the users using the terminal devices 501, 502, and 503. The background management server may store and process the received data generated by the client in a data source, and feed back the data storage result to the terminal device.

[0096] It should be noted that the method for synchronizing data provided in the embodiment of the present invention is generally executed by one or more servers 505. Accordingly, the device for synchronizing data is generally set in one or more servers 505, wherein the log processing layer and the synchronization business layer in the embodiment of the present invention can belong to different servers or to the same server.

[0097] It should be understood that Figure 5 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0098] Reference below Figure 6 , which shows a schematic diagram of the structure of a computer system 600 of a terminal device suitable for implementing an embodiment of the present invention. Figure 6 The terminal device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0099] like Figure 6 As shown, the computer system 600 includes a central processing unit (CPU) 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage part 608 into a random access memory (RAM) 603. In the RAM 603, various programs and data required for the operation of the system 600 are also stored. The CPU 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0100] The following components are connected to the I / O interface 605: an input section 606 including a keyboard, a mouse, etc.; an output section 607 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 608 including a hard disk, etc.; and a communication section 609 including a network interface card such as a LAN card, a modem, etc. The communication section 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to the I / O interface 605 as needed. A removable medium 611, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 610 as needed, so that a computer program read therefrom is installed into the storage section 608 as needed.

[0101] In particular, according to the embodiments disclosed in the present invention, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present invention include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 609, and / or installed from the removable medium 611. When the computer program is executed by the central processing unit (CPU) 601, the above-mentioned functions defined in the system of the present invention are executed.

[0102] It should be noted that the computer-readable medium shown in the present invention may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present invention, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0103] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0104] The modules and / or units involved in the embodiments of the present invention may be implemented in software or hardware. The modules and / or units described may also be set in a processor, for example, they may be described as: a processor includes a log processing module and a data synchronization module; wherein the names of these modules do not constitute a limitation on the modules themselves in certain circumstances, for example, the log processing module may also be described as "a module for filtering out target log information that matches a preset filtering policy from the data log".

[0105] As another aspect, the present invention also provides a computer-readable medium, which may be included in the device described in the above embodiment; or it may exist independently without being assembled into the device. The above computer-readable medium carries one or more programs, and when the above one or more programs are executed by a device, the device includes: collecting data logs from multiple first data sources, filtering out target log information matching a preset filtering policy from the data logs, and storing the target log information; obtaining the stored target log information with changes through a preset subscription service; parsing the change information from the target log information with changes, and synchronizing the data to be synchronized corresponding to the change information to the second data source.

[0106] In the embodiment of the present invention, target log information matching a preset filtering strategy is filtered out from data logs collected from multiple first data sources, and the target log information is obtained through a preset subscription service; change information is parsed from the target log information, and the data to be synchronized corresponding to the change information is synchronized to a second data source. By filtering the original data logs accordingly and decoupling the processing of the filtered logs from the synchronized data through the subscription service, the efficiency of synchronized data is improved and the resources consumed by synchronized data are reduced.

[0107] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may occur depending on design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for synchronizing data, characterized in that: include: Collecting data logs from a plurality of first data sources, filtering out target log information matching a preset filtering policy from the data logs, and storing the target log information; Obtain stored target log information with changes through the preset subscription service; The change information is parsed from the target log information with changes, and the to-be-synchronized data corresponding to the change information is synchronized to the second data source.

2. The method according to claim 1, characterized in that Also includes: At least one log processing layer and at least one synchronization service layer are decoupled from each other, wherein each of the log processing layers is configured with a preset filtering strategy corresponding to the synchronization service layer, and each of the synchronization service layers is bound to at least one of the subscription services; Execute the step of filtering out target log information matching a preset filtering strategy from the data log by any idle log processing layer; The synchronization service layer executes the steps of acquiring stored target log information with changes and parsing change information from the target log information with changes.

3. The method according to claim 1 or 2, characterized in that: The preset filtering strategy includes data identification filtering, data processing instruction filtering, regular expression filtering, multiple filtering types in the custom filtering strategy and the execution order of the multiple filtering types; The filtering out target log information matching a preset filtering strategy from the data log includes: Parsing out the multiple filtering types included in the preset filtering strategy and the execution order of the multiple filtering types; According to the execution order of the plurality of filter types, target log information matching each of the parsed filter types is filtered out from the data log.

4. The method according to claim 1, characterized in that Further including: Building an index for the log file corresponding to the collected data log, and generating an index file for the log file; In response to searching for the data log or the log file of the target log information, the log file is located through the index file.

5. The method according to claim 4, characterized in that Further including: Searching for a log file corresponding to the data log or the target log information that satisfies a set cleaning strategy; If found, clean up the log files that meet the set cleanup strategy; The index file is updated.

6. The method according to claim 4 or 5, characterized in that: Also includes: Obtaining a synchronization result of the second data source for the data to be synchronized; In the case where the synchronization result indicates a synchronization failure, according to the failure information corresponding to the synchronization failure indicated by the synchronization result, locating a log file associated with the failure information according to the index file; Target log information matching the preset filtering policy is re-filtered from the log file associated with the failure information.

7. The method according to claim 2, characterized in that Also includes: When the target log information is written into a new data log file, compressing the new log file; The acquiring of stored target log information having changes includes: Get the target log information with changes from the compressed new log file.

8. A device for synchronizing data, characterized in that: include: Processing log module and synchronization data module; among them, The log processing module is used to collect data logs from multiple first data sources, filter out target log information matching a preset filtering strategy from the data logs, and store the target log information; The synchronization data module is used to obtain the stored target log information with changes through a preset subscription service; parse the change information from the target log information with changes, and synchronize the data to be synchronized corresponding to the change information to the second data source.

9. An electronic device, characterized in that: include: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 7.

10. A computer readable medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images