Data synchronization method and related device

Abstract

The invention provides a data synchronization method and a related device, and the method comprises the steps: obtaining the change information of a business source, and writing the change information into message middleware; and obtaining to-be-processed data of the message middleware, adjusting the data format of the to-be-processed data to a standardized data format to obtain target data, and sending the target data to the second data platform to realize data synchronization. Therefore, based on a streaming real-time synchronization technology, the service data is synchronized to the data warehouse in real time, and the data is processed in real time, peak clipping and valley filling are carried out, so that the pressure of network bandwidth is reduced, and the data synchronization efficiency is improved.

Description

Technical Field

[0001] The present application belongs to the field of data processing technology, and in particular, relates to a data synchronization method and related devices. Background Art

[0002] With the rapid development of the information age, the amount of data accumulated by enterprises has exploded. The accumulation of historical data and low frequency of use have led to the decline of business database performance. At the same time, the independent data extraction systems of various departments have also led to inconsistent data and waste of resources. In order to solve the above problems, data warehouses came into being.

[0003] Existing ETL (Extract-Transform-Load) technology is mainly offline, mainly through batch query, to synchronize data to the data warehouse. Offline data retrieval will put pressure on the business database, have high requirements for network bandwidth, and is prone to peaks and troughs, which may cause network instability. Summary of the invention

[0004] The embodiments of the present application provide a data synchronization method and related devices, which collect real-time changing business data and synchronize it to a data warehouse, perform real-time processing on the data, achieve peak shaving and valley filling, reduce the pressure on network bandwidth, and improve data synchronization efficiency.

[0005] In a first aspect, an embodiment of the present application provides a data synchronization method, which is applied to a first data platform, and the method includes:

[0006] Obtain change information of the business source and write the change information into the message middleware. The change information is used to indicate the real-time business data generated in the business source.

[0007] Get the data to be processed by the message middleware. The data to be processed is the data formed after the message middleware decouples the change information from the business.

[0008] Adjusting the data format of the data to be processed into a standardized data format to obtain target data;

[0009] The target data is sent to the second data platform to achieve data synchronization. The second data platform refers to a data system for storing data in batches.

[0010] Among them, the data format of the data to be processed is adjusted to a standardized data format to obtain the target data, including: determining the format of the data to be processed, converting the format of the data to be processed to obtain the first structured data; performing a cleaning operation on the first structured data to obtain the second structured data, and the cleaning operation includes at least one of the following operations: filtering, cleaning, standardization processing and stream connection processing; integrating the second structured data to obtain the target data.

[0011] Among them, after obtaining the change information of the business source, the method also includes: presetting a starting timestamp, the starting timestamp is used to characterize the verifiable data that exists at a specific time for the data to be processed; and determining the data center identifier of the business source of the data to be processed; determining the current timestamp, subtracting the starting timestamp from the current timestamp to obtain a timestamp identifier; generating a serial number of the current millisecond; and generating a globally unique identifier for the data to be processed based on the timestamp identifier, the serial number, the data center identifier and the machine identifier.

[0012] Among them, after generating the globally unique identifier of the data to be processed, the method also includes: obtaining task data after executing batching for the data in the first data table; executing batching and warehousing operations according to the task data, and sending them to the data warehouse analysis tool; determining the target primary key of the data to be processed through the data warehouse analysis tool, and locating the record of the data to be processed through the target primary key, the target primary key being an identifier that uniquely represents the record in the database table; sending the globally unique identifier of the data to be processed to the second data table, so that the globally unique identifier of the task data is automatically matched when the task data is started subsequently.

[0013] The method also includes: configuring a first custom parameter for the data to be processed, and obtaining the business time data of the current batch of data, the business time data is used to characterize the earliest business time of the current batch of data; according to the business time data, obtaining the first target incremental data of the current batch of data, the target incremental data refers to the data that needs to be processed; and storing it in the detailed data layer.

[0014] Among them, after storing in the detailed data layer, the method also includes: configuring a second custom parameter, and obtaining the business time data of the current batch of data in one or more operation data storage tables upstream of the current batch of data; according to the business time data, obtaining the second target increment of the current batch of data; based on the second target increment, updating to the label table.

[0015] The method also includes: after obtaining the data to be processed, analyzing the business sources of the data to be processed, the number of business sources includes one or more; collecting the business source logs of the business sources, and cleaning the business source logs to remove irrelevant information in the business source logs; and converting the different formats of the cleaned business source logs into a unified format to obtain a target log; parsing the target log to determine the system operation status of the business source to monitor the business source in real time.

[0016] In a second aspect, a data synchronization device is applied to a first data platform, and the device includes: a first acquisition unit, a second acquisition unit, an adjustment unit and a synchronization unit; wherein the first acquisition unit is used to obtain change information of a business source and write the change information into a message middleware, and the change information is used to indicate real-time business data generated in the business source; the second acquisition unit is used to obtain the data to be processed of the message middleware, and the data to be processed is the data formed after the message middleware decouples the change information from the business; the adjustment unit is used to adjust the data format of the data to be processed to a standardized data format to obtain the target data; the synchronization unit is used to send the target data to the second data platform to achieve data synchronization, and the second data platform refers to a data system for storing data in batches.

[0017] In a third aspect, an embodiment of the present application provides an electronic device, comprising: a processor and a memory; and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the processor, and the program includes instructions for some or all of the steps described in the first aspect.

[0018] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium is used to store a computer program, wherein the computer program enables a computer to execute instructions of some or all of the steps described in the first aspect of the embodiment of the present application.

[0019] In a fifth aspect, an embodiment of the present application provides a computer program product, wherein the computer program product includes a non-transitory computer-readable storage medium storing a computer program, and the computer program is operable to cause a computer to execute some or all of the steps described in the first aspect of the embodiment of the present application. The computer program product may be a software installation package.

[0020] It can be seen that the embodiment of the present application provides a data synchronization method and related devices. The method first obtains the change information of the business source and writes the change information into the message middleware. The change information is used to indicate the real-time business data generated in the business source; secondly, the data to be processed of the message middleware is obtained. The data to be processed is the data formed after the message middleware decouples the change information from the business; then, the data format of the data to be processed is adjusted to a standardized data format to obtain the target data; finally, the target data is sent to the second data platform to achieve data synchronization. The second data platform refers to a data system for batch storage of data. In this way, based on the streaming real-time synchronization technology, the business data is synchronized to the data warehouse in real time. By processing the data in real time, the peak is cut and the valley is filled, the pressure on the network bandwidth is reduced, and the data synchronization efficiency is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0022] Figure 1 is a system schematic diagram of a data synchronization method provided in an embodiment of the present application;

[0023] Figure 2 It is a specific flow chart of a data synchronization method provided in an embodiment of the present application;

[0024] Figure 3 This is a schematic diagram of a data synchronization architecture provided by an embodiment of the present application;

[0025] Figure 4 It is a schematic diagram of another data synchronization architecture provided by an embodiment of the present application;

[0026] Figure 5 This is a schematic diagram of a real-time calculation process of data synchronization provided by an embodiment of the present application;

[0027] Figure 6 It is a block diagram of the functional units of a data synchronization device provided in an embodiment of the present application;

[0028] Figure 7 It is a structural block diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0029] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0030] The terms "first", "second", etc. in the specification and claims of this application and the above-mentioned drawings are used to distinguish different objects, rather than to describe a specific order. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but optionally includes steps or units that are not listed, or optionally includes other steps or units inherent to these processes, methods, products or devices.

[0031] It should be understood that the term "and / or" in this article is only a description of the association relationship of the associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article indicates that the associated objects before and after are in an "or" relationship.

[0032] The "multiple" in the embodiments of the present application refers to two or more. The "connection" in the embodiments of the present application refers to various connection modes such as direct connection or indirect connection to achieve communication between devices, and the embodiments of the present application do not make any limitation on this.

[0033] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0034] Before further describing the embodiments of the present disclosure in detail, the nouns and terms involved in the embodiments of the present disclosure are described. The nouns and terms involved in the embodiments of the present disclosure are applicable to the following interpretations:

[0035] Data warehouse: A data warehouse is a subject-oriented, integrated, relatively stable data set that reflects historical changes and is used to support management decisions. Real-time data warehouses are mainly used to solve the problem of low timeliness of traditional data warehouses. They are often used in real-time OLAP analysis, real-time data dashboards, and real-time monitoring of business indicators.

[0036] CDC (Change Data Capture) is a technology used to capture changes in data sources. It can capture insert, update, and delete operations in the source system and capture these changed data into the CDC mechanism. Enabling CDC in the data warehouse can achieve incremental data synchronization between the data warehouse and various data sources, improving the real-time performance and accuracy of the data warehouse.

[0037] Existing ETL (Extract-Transform-Load) technology is mainly offline, mainly through batch query, to synchronize data to the data warehouse. Offline data retrieval will put pressure on the business database, have high requirements for network bandwidth, and is prone to peaks and troughs, which may cause network instability.

[0038] In response to the above problems, the embodiments of the present application provide a data synchronization method and related devices, which collect real-time changing business data and synchronize it to a data warehouse, perform real-time data processing on the data, achieve peak shaving and valley filling, reduce the pressure on network bandwidth, and improve data synchronization efficiency.

[0039] Please refer to Figure 1 , Figure 1 is a system diagram of a data synchronization method provided by an embodiment of the present application, such as Figure 1 As shown, the system may include a first terminal 110, a second terminal 120, a third terminal 130, a network 140, a first data platform 150, and a second data platform 160. The network 140 is used to provide a medium for a communication link between the first terminal 110, the second terminal 120, the third terminal 130, and the first data platform 150. The network 140 may include various connection types, such as wired, wireless communication links, or optical fiber cables, etc.

[0040] The first terminal 110, the second terminal 120, and the third terminal 130 interact with the first data platform 150 through the network 140 to receive or send messages, etc. Various business client applications may be installed on the first terminal 110, the second terminal 120, and the third terminal 130; the first terminal 110, the second terminal 120, and the third terminal 130 may be various electronic devices with display screens and supporting web browsing, including but not limited to smart phones, tablet computers, laptop portable computers, and desktop computers, etc.

[0041] The first data platform 150 can be a cloud digital warehouse, also known as a cloud data warehouse or cloud data warehouse, which is a data storage and analysis system based on cloud computing technology. It uses the computing resources, storage resources and data services of the cloud computing platform to provide enterprises and organizations with efficient, flexible and scalable data management capabilities. The cloud digital warehouse focuses on processing and analyzing large-scale data sets, supports complex data query, report generation, data mining and machine learning tasks, and feeds back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal.

[0042] The second data platform 160 may be an offline digital warehouse, which is a system specifically used to store, manage and analyze a large amount of historical data. It uses batch processing to update data regularly (such as daily, weekly, and monthly) to support various business analysis, report generation, and data mining tasks.

[0043] In an embodiment of the present application, first, the change information of the business source is obtained and written into the message middleware, the change information is used to indicate the real-time business data generated in the business source; secondly, the data to be processed of the message middleware is obtained, and the data to be processed is the data formed after the message middleware decouples the change information from the business; then, the data format of the data to be processed is adjusted to a standardized data format to obtain the target data; finally, the target data is sent to the second data platform to achieve data synchronization, and the second data platform refers to a data system used to store data in batches.

[0044] A data synchronization method provided by an embodiment of the present application is introduced below.

[0045] See also Figure 2 , Figure 2 is a specific flow chart of a data synchronization method provided in an embodiment of the present application, and the method is applied to Figure 1 The first data platform 150 shown includes the following steps:

[0046] Step S210: Obtain change information of the service source and write the change information into the message middleware.

[0047] The change information is used to indicate the real-time business data generated in the business source.

[0048] Among them, the change information of the source database is obtained based on the Flink CDC technology, and the change information is written into the message middleware; the data to be processed is obtained based on the message middleware. In this embodiment, the incremental change records of the source database can be synchronized to one or more data destinations (Sink) using the Flink CDC (Change Data Capture) technology. During the synchronization process, the data can also be processed to a certain extent, such as grouping (GROUP BY), association of multiple tables (JOIN), etc., so that each independent data set is analyzed without affecting each other, while maintaining real-time synchronization with the upstream data.

[0049] The data change information includes data insertion (create), update (update) or deletion (delete), etc. The message middleware can be a Kafka database.

[0050] Step S220, obtaining the data to be processed by the message middleware.

[0051] The data to be processed is the data formed after the message middleware performs business decoupling on the change information. The Kafka database has the function of business decoupling.

[0052] Step S230, adjusting the data format of the data to be processed to a standardized data format to obtain target data.

[0053] In one possible example, the data format of the data to be processed is adjusted to a standardized data format to obtain target data, including: determining the format of the data to be processed, converting the format of the data to be processed to obtain first structured data; performing a cleaning operation on the first structured data to obtain second structured data, the cleaning operation including at least one of the following operations: filtering, cleaning, standardization processing and stream connection processing; integrating the second structured data to obtain target data.

[0054] The first data platform may further include a data source layer ODS, a data detail layer DWD, a data service layer DWS and a data application layer ADS.

[0055] Step S240: Send target data to the second data platform to achieve data synchronization.

[0056] The second data platform refers to a data system used to store data in batches, which is also an offline digital warehouse.

[0057] It can be seen that the embodiment of the present application provides a data synchronization method and related devices. The method first obtains the change information of the business source and writes the change information into the message middleware. The change information is used to indicate the real-time business data generated in the business source; secondly, the to-be-processed data of the message middleware is obtained. The to-be-processed data is the data formed after the message middleware decouples the change information from the business; then, the data format of the to-be-processed data is adjusted to a standardized data format to obtain the target data; finally, the target data is sent to the second data platform to achieve data synchronization. The second data platform refers to a data system for batch storage of data. In this way, based on the streaming real-time synchronization technology, the business data is synchronized to the data warehouse in real time. By processing the data in real time, the peak is cut and the valley is filled, the pressure on the network bandwidth is reduced, and the data synchronization efficiency is improved.

[0058] In one possible example, after obtaining the change information of the business source, the method also includes: presetting a starting timestamp, the starting timestamp is used to characterize the verifiable data that exists at a specific moment in the data to be processed; and determining the data center identifier of the business source of the data to be processed; determining the current timestamp, subtracting the starting timestamp from the current timestamp to obtain a timestamp identifier; generating a serial number of the current millisecond; and generating a globally unique identifier for the data to be processed based on the timestamp identifier, the serial number, the data center identifier and the machine identifier.

[0059] Among them, a Snowflake algorithm that generates a globally unique ID in a distributed system can be used. The composition structure of the Snowflake ID The ID generated by Snowflake consists of the following parts: Unused bit: 1 bit. Since the long type is signed in languages ​​such as Java, the highest bit is the sign bit, the positive number is 0, and the negative number is 1. The generated IDs are all positive numbers, so this bit is fixed to 0; timestamp: 41 bits, used to record the timestamp (millisecond level); data center ID: 5 bits, used to record the data center ID; machine ID (work node ID): 5 bits, used to record the work machine ID; serial number: 12 bits, used to record different IDs generated within the same millisecond.

[0060] See also Figure 3 , Figure 3 is a schematic diagram of a data synchronization architecture provided by an embodiment of the present application. Figure 3 It includes: a first MySQL library 301 and a second MySQL library 302, wherein the subsequent task indicated by the first MySQL library 301 is Flink task one 303, and the subsequent task indicated by the second MySQL library 302 is Flink task two 304; and the first MySQL library 301 is a Las library, which is a database specially used to store specific types (such as LiDAR data), and the second MySQL library 302 is a casv2db library, which is another custom or specific database used to store different data types.

[0061] Specifically, Flink task one 303: reads data from the Las library; uses the database import command to transfer the data from the Las library to the first Mysql library 301; pushes the processed data (which may have been in the first Mysql library 301 or processed directly from the Las library) to a first Kafka message set 305 in real time through the first Kafka connector 3031 and the second Kafka connector 3032.

[0062] Flink Task 2: Similarly, data is read from the casv2db library; similarly, the data may be transferred to the second Mysql library 302 through a database import command; the processed data is also pushed to another second Kafka message set 306 in real time, or pushed to the same topic as Flink Task 1, depending on the specific design of the system.

[0063] Among them, the database import command is the source or component that reads data from the first Mysql library 301 and the second Mysql library 302. It is responsible for extracting data from the Mysql library for further processing; the first Kafka connector 3031, the second Kafka connector 3032, the third Kafka connector 3041 and the fourth Kafka connector 3042 are a component or operation that writes data to Kafka. Here, it is used to push data to the Kafka message set in real time; the first Kafka message set 305 and the second Kafka message set 306 are a publish-subscribe message queue for storing data records. Data producers publish messages to a specific topic, while data consumers subscribe to and read messages from the topic; real-time push refers to the real-time nature of data processing, that is, once the data is generated or modified, it is immediately processed and pushed to the downstream system with almost no delay; it realizes the orderly identification of multiple states of data to ensure accurate and orderly update of data.

[0064] It can be seen that in this example, by presetting the start timestamp, determining the data center identifier of the business source of the data to be processed, determining the current timestamp, subtracting the start timestamp from the current timestamp to obtain the timestamp identifier, generating the sequence number of the current millisecond, and generating the global unique identifier of the data to be processed based on the timestamp identifier, the sequence number, the data center identifier and the machine identifier. In this way, each record in the system is uniquely identified to ensure the consistency and integrity of the data.

[0065] In one possible example, after generating a globally unique identifier for the data to be processed, the method further includes: performing batching on the data in the first data table to obtain task data; performing batching and warehousing operations according to the task data, and sending the data to a data warehouse analysis tool; determining the target primary key of the data to be processed through the data warehouse analysis tool, and locating the record of the data to be processed through the target primary key, wherein the target primary key is an identifier that uniquely represents the record in the database table; and sending the globally unique identifier of the data to be processed to the second data table, so that the globally unique identifier of the task data is automatically matched when the task data is subsequently started.

[0066] Please refer to Figure 4 , Figure 4 is a schematic diagram of another data synchronization architecture provided by an embodiment of the present application, such as Figure 4As shown, the system first accesses the third MySQL library 401, which is used to record the auto-increment ID of the task, and obtains the next available auto-increment ID from the third MySQL library 401 as the unique identifier of the current task; when the task is completed and the data is ready to be submitted, the system will write the processed data to the Hive database 404, which is a data warehouse tool built on Hadoop and is used to query and manage large data sets stored in the Hadoop distributed file system. It provides a query language HiveQL similar to SQL, allowing users to easily analyze and process data; in the entire process, Sink node 1 402 and sink node 2 403 play an important role, which is responsible for processing and recording the data transfer process from MySQL to Hive, including the start of the task, the writing of data, and the update of the auto-increment ID. Sink node 1 402 and sink node 2 403 can be a custom application or service specifically used to handle such data transfer tasks.

[0067] See also Figure 4 In step S4041, after the batch is stored in the warehouse, the current self-increment ID is recorded.

[0068] When the data is written into the Hive database 404 by batch storage, the current auto-increment ID is recorded. The auto-increment ID is an automatically increasing unique identifier, which is usually used in a database table to uniquely identify each record. In this process, the auto-increment ID is used to uniquely identify each task.

[0069] Step S4042, obtain the self-incrementing ID of the task.

[0070] The system will return the currently used auto-increment ID to the table of the third MySQL database 401 so that it can be used when the next task is started.

[0071] The batch storage of data means that the pre-processed data will be stored in the database in batches. This process may involve the design of database tables, field mapping, and data insertion operations. The advantages of batch storage are that it can reduce the number of database operations, improve the efficiency of data processing, and help maintain data consistency and integrity.

[0072] It can be seen that in this example, the task data is obtained by executing batch aggregation on the data in the first data table; the batch aggregation operation is performed according to the task data and sent to the data warehouse analysis tool; the target primary key of the data to be processed is determined by the data warehouse analysis tool, and the record of the data to be processed is located by the target primary key; the global unique identifier of the data to be processed is sent to the second data table, so that the global unique identifier of the task data can be automatically matched when the task data is started subsequently. In this way, the real-time storage capability of variable increments is realized, providing a basic storage base for the data warehouse.

[0073] See also Figure 5 , Figure 5 is a schematic diagram of a real-time calculation process of data synchronization provided by an embodiment of the present application, such as Figure 5 As shown, the following steps are included:

[0074] Step S510: configure the first custom parameter for the data to be processed, and obtain the business time data of the current batch in the table.

[0075] The business time data is used to represent the earliest business time in the current batch.

[0076] Step S520, obtaining the first target incremental data of the current batch according to the business time data.

[0077] Among them, the target incremental data refers to the data that needs to be processed.

[0078] Step S530: store to the detailed data layer.

[0079] Among them, the Detail Data Layer is also commonly referred to as DWD (Data Warehouse Detail), which is the level in the data warehouse used to store the detailed data after cleaning, conversion and loading. These data maintain the same or similar structure and granularity as the business system data, and are the interface between the data warehouse analysis system data and the system data.

[0080] Step S540, configure the second custom parameter, and obtain the business time data of the current batch of data in one or more upstream operation data storage tables of the current batch of data.

[0081] The business time data is used to represent the earliest business time in the current batch.

[0082] Step S550, obtaining the second target increment of the current batch according to the business time data.

[0083] Step S560: Update the tag table based on the second target increment.

[0084] In a possible example, the method also includes: after obtaining the data to be processed, analyzing the business sources of the data to be processed, the number of business sources includes one or more; collecting business source logs of the business sources, and cleaning the business source logs to remove irrelevant information in the business source logs; and converting different formats of the cleaned business source logs into a unified format to obtain a target log; parsing the target log to determine the system operation status of the business source to monitor the business source in real time.

[0085] Specifically, first, it is necessary to determine which business systems or business modules the data to be processed comes from. The number of business sources may be one or more, depending on the complexity and diversity of the data. The purpose of analyzing business sources is to understand the source and background of the data so that targeted processing and analysis can be carried out later; then, business source logs are an important data source for recording information such as the operating status of business systems, user behaviors, and system errors. By collecting these logs, rich business data can be obtained for subsequent analysis and monitoring. Methods for collecting logs may include real-time monitoring of log files, periodic downloading of log files, and the use of log collection tools.

[0086] Next, since the business source logs may contain a large amount of irrelevant information, redundant data, or noise data, they need to be cleaned. The purpose of cleaning logs is to remove these irrelevant information and retain data that is useful for subsequent analysis. The cleaning process may include removing duplicate records, filtering invalid data, correcting erroneous data, etc.; since different business sources may use different log formats, in order to facilitate subsequent processing and analysis, these logs in different formats need to be converted into a unified format. Logs in a unified format can simplify the data processing process, improve processing efficiency, and facilitate subsequent data integration and query.

[0087] Then, after cleaning and format conversion, the target log is a data set with a clear structure and unified format. Next, these logs need to be parsed to extract useful information, such as system operation status, user behavior patterns, abnormal errors, etc. By analyzing this information, the system operation status of the business source can be determined, including whether the system is running normally, whether there are potential problems or risks, etc.

[0088] Finally, based on the parsed target logs, a real-time monitoring mechanism can be established to continuously monitor the business source. Real-time monitoring can promptly detect abnormal conditions and potential problems in the system and trigger corresponding early warning and alarm mechanisms. This helps to respond to problems quickly, reduce losses, and ensure the stable operation of the business system.

[0089] It can be seen that in this example, after obtaining the data to be processed, the business source of the data to be processed is analyzed; the business source logs of the business source are collected, and the business source logs are cleaned to remove irrelevant information in the business source logs; and the different formats of the cleaned business source logs are converted into a unified format to obtain the target logs; the target logs are parsed to determine the system operation status of the business source to monitor the business source in real time. In this way, based on the streaming real-time synchronization technology, the business data is synchronized to the data warehouse in real time, which can improve the accuracy and reliability of the data.

[0090] The above mainly introduces the scheme of the embodiment of the present application from the perspective of the execution process on the method side. It is understandable that in order to realize the above functions, the mobile electronic device includes a hardware structure and / or software module corresponding to the execution of each function. Those skilled in the art should easily realize that, in combination with the units and algorithm steps of each example described in the embodiment provided in this article, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0091] The embodiment of the present application can divide the electronic device into functional units according to the above method example. For example, each functional unit can be divided according to each function, or two or more functions can be integrated into one processing unit. The above integrated unit can be implemented in the form of hardware or in the form of software functional units. It should be noted that the division of units in the embodiment of the present application is schematic and is only a logical function division. There may be other division methods in actual implementation.

[0092] and Figure 2 For details, see Figure 6 , Figure 6 is a block diagram of the functional units of a data synchronization device provided in an embodiment of the present application, such as Figure 6As shown, the data synchronization device 600 includes: a first acquisition unit 610, a second acquisition unit 620, an adjustment unit 630 and a synchronization unit 640; wherein the first acquisition unit 610 is used to obtain change information of a business source and write the change information into a message middleware, the change information is used to indicate real-time business data generated in the business source; the second acquisition unit 620 is used to obtain the data to be processed of the message middleware, the data to be processed is the data formed after the message middleware performs business decoupling on the change information; the adjustment unit 630 is used to adjust the data format of the data to be processed to a standardized data format to obtain target data; the synchronization unit 640 is used to send the target data to a second data platform to achieve data synchronization, the second data platform refers to a data system for storing data in batches.

[0093] In one possible example, the data format of the data to be processed is adjusted to a standardized data format to obtain target data, and the adjustment unit 630 is specifically used to: determine the format of the data to be processed, convert the format of the data to be processed, and obtain first structured data; perform a cleaning operation on the first structured data to obtain second structured data, and the cleaning operation includes at least one of the following operations: filtering, cleaning, standardization processing, and stream connection processing; integrate the second structured data to obtain target data.

[0094] In one possible example, after obtaining the change information of the business source, the first acquisition unit 610 is specifically used to: preset a starting timestamp, which is used to represent verifiable data that exists at a specific moment in the data to be processed; and determine the data center identifier of the business source of the data to be processed; determine the current timestamp, subtract the starting timestamp from the current timestamp to obtain a timestamp identifier; generate a serial number of the current millisecond; and generate a globally unique identifier for the data to be processed based on the timestamp identifier, the serial number, the data center identifier and the machine identifier.

[0095] In one possible example, after generating the globally unique identifier of the data to be processed, the first acquisition unit 610 is specifically used to: obtain task data after executing batch aggregation for the data in the first data table; execute batch aggregation and storage operations according to the task data, and send it to the data warehouse analysis tool; determine the target primary key of the data to be processed through the data warehouse analysis tool, and locate the record of the data to be processed through the target primary key, the target primary key is an identifier that uniquely represents the record in the database table; send the globally unique identifier of the data to be processed to the second data table, so that the globally unique identifier of the task data can be automatically matched when the task data is started subsequently.

[0096] In a possible example, the data synchronization device 600 is also specifically used to: configure a first custom parameter for the data to be processed, and obtain business time data of the current batch of data, the business time data is used to characterize the earliest business time of the current batch of data; according to the business time data, obtain the first target incremental data of the current batch of data, the target incremental data refers to the data that needs to be processed; and store it in the detailed data layer.

[0097] In one possible example, after storing in the detailed data layer, the data synchronization device 600 is specifically used to: configure a second custom parameter, and obtain the business time data of the current batch of data in one or more operation data storage tables upstream of the current batch of data; obtain the second target increment of the current batch of data based on the business time data; and update the label table based on the second target increment.

[0098] In a possible example, the data synchronization device 600 is specifically used to: after acquiring the data to be processed, analyze the business sources of the data to be processed, the number of business sources includes one or more; collect the business source logs of the business sources, and clean the business source logs to remove irrelevant information in the business source logs; and convert different formats of the cleaned business source logs into a unified format to obtain a target log; parse the target log to determine the system operation status of the business source to monitor the business source in real time.

[0099] It can be understood that since the method embodiment and the device embodiment are different presentation forms of the same technical concept, the content of the method embodiment part in this application should be synchronously adapted to the device embodiment part and will not be repeated here.

[0100] Figure 7 is a structural block diagram of an electronic device provided in an embodiment of the present application. Figure 7 As shown, the electronic device 700 may include one or more of the following components: a processor 701, a memory 702 coupled to the processor 701, wherein the memory 702 may store one or more computer programs, and the one or more computer programs may be configured to implement the methods described in the above examples when executed by one or more processors 701. Figure 7 It can be the above Figure 1 The first terminal 110, the second terminal 120, or the third terminal 130.

[0101] The processor 701 may include one or more processing cores. The processor 701 uses various interfaces and lines to connect the various parts of the entire electronic device 700, and executes various functions and processes data of the electronic device 700 by running or executing instructions, programs, code sets or instruction sets stored in the memory 702, and calling data stored in the memory 702. Optionally, the processor 701 can be implemented in at least one hardware form of digital signal processing (Digital Signal Processing, DSP), field programmable gate array (Field-Programmable Gate Array, FPGA), and programmable logic array (Programmable Logic Array, PLA). The processor 701 can integrate one or a combination of a central processing unit (Central Processing Unit, CPU), an image processor (Graphics Processing Unit, GPU) and a modem. It is understandable that the above-mentioned modem may not be integrated into the processor 701, and may be implemented separately through a communication chip.

[0102] The memory 702 may include a random access memory (RAM) or a read-only memory (ROM). The memory 702 may be used to store instructions, programs, codes, code sets or instruction sets. The memory 702 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for implementing at least one function (such as a touch function, a sound playback function, an image playback function, etc.), instructions for implementing the above-mentioned various method examples, etc. The data storage area may also store data created by the electronic device 700 during use, etc.

[0103] It is understandable that the electronic device 700 may include more or fewer structural elements than those in the above structural block diagram, for example, a power module, physical buttons, a WiFi (Wireless Fidelity) module, a speaker, a Bluetooth module, a sensor, etc., which are not limited here.

[0104] An embodiment of the present application also provides a computer storage medium, on which a computer program / instruction is stored. When the computer program / instruction is executed by a processor, part or all of the steps of any method recorded in the above method embodiments are implemented.

[0105] An embodiment of the present application also provides a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program, and the computer program is operable to cause a computer to execute part or all of the steps of any method recorded in the above method embodiments.

[0106] It should be understood that in the various embodiments of the present application, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0107] In the several embodiments provided in the present application, it should be understood that the disclosed methods, devices and systems can be implemented in other ways. For example, the device embodiments described above are only schematic; for example, the division of units is only a logical function division, and there may be other division methods in actual implementation; for example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0108] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0109] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may be physically included separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0110] The above-mentioned integrated unit implemented in the form of a software functional unit can be stored in a computer-readable storage medium. The above-mentioned software functional unit is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform some steps of the methods of various embodiments of the present invention. The aforementioned storage medium includes: a USB flash drive, a mobile hard disk, a magnetic disk, an optical disk, a volatile memory or a non-volatile memory. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM) or a flash memory. The volatile memory can be a random access memory (RAM), which is used as an external cache. By way of example but not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic random access memory (DRAM), synchronous DRAM (SDRAM), double data rate synchronous dynamic random access memory (double data rate SDRAM, DDR SDRAM), enhanced synchronous dynamic random access memory (enhanced SDRAM, ESDRAM), synchronous link dynamic random access memory (synchlink DRAM, SLDRAM) and direct memory bus random access memory (directrambus RAM, DR RAM) and other media that can store program code.

[0111] Although the present invention is disclosed as above, the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions without departing from the spirit and scope of the present invention, and can make various changes and modifications, including the combination of the above-mentioned different functions and implementation steps, including software and hardware implementation methods, all of which are within the scope of protection of the present invention.

Claims

1. A data synchronization method, characterized in that: Applied to the first data platform, the method includes: Acquire change information of a service source, and write the change information into a message middleware, wherein the change information is used to indicate real-time service data generated in the service source; Acquire the data to be processed of the message middleware, where the data to be processed is data formed after the message middleware performs business decoupling on the change information; Adjusting the data format of the data to be processed to a standardized data format to obtain target data; The target data is sent to a second data platform to achieve data synchronization, where the second data platform refers to a data system for storing data in batches.

2. The method according to claim 1, characterized in that The step of adjusting the data format of the data to be processed to a standardized data format to obtain target data includes: Determine the format of the data to be processed, and convert the format of the data to be processed to obtain first structured data; Performing a cleaning operation on the first structured data to obtain second structured data, wherein the cleaning operation includes at least one of the following operations: filtering, cleaning, standardization processing, and stream connection processing; The second structured data is integrated to obtain the target data.

3. The method according to claim 1 or 2, characterized in that: After acquiring the change information of the service source, the method further includes: Preset a start timestamp, the start timestamp is used to represent the verifiable data that the data to be processed exists at a specific time; and determine the data center identifier of the business source of the data to be processed; Determine a current timestamp, and subtract the start timestamp from the current timestamp to obtain a timestamp identifier; Generate the current millisecond serial number; A globally unique identifier of the data to be processed is generated according to the timestamp identifier, the serial number, the data center identifier and the machine identifier.

4. The method according to claim 3, characterized in that After generating the globally unique identifier of the data to be processed, the method further includes: Execute batching for the data in the first data table to obtain task data; Execute batch storage operation according to the task data and send it to the data warehouse analysis tool; Determine, by means of the data warehouse analysis tool, a target primary key of the data to be processed, and locate the record of the data to be processed by means of the target primary key, wherein the target primary key is an identifier that uniquely represents a record in a database table; The globally unique identifier of the data to be processed is sent to the second data table, so that the globally unique identifier of the task data is automatically matched when the task data is subsequently started.

5. The method according to claims 1-4, characterized in that: The method further comprises: Configure a first custom parameter for the data to be processed, and obtain business time data in the current batch of data, where the business time data is used to characterize the earliest business time in the current batch of data; According to the business time data, first target incremental data of the current batch is obtained, where the target incremental data refers to data that needs to be processed; Stored in the detail data layer.

6. The method according to claim 5, characterized in that After storing in the detailed data layer, the method further includes: Configure the second custom parameter, and obtain the business time data of the current batch of table drops in one or more operation data storage tables upstream of the current batch of table drops; According to the business time data, a second target increment of the current batch of items is obtained; Based on the second target increment, update the tag table.

7. The method according to any one of claims 1 to 6, characterized in that: The method further comprises: After acquiring the data to be processed, analyzing the business sources of the data to be processed, wherein the number of the business sources includes one or more; The business source logs of the business source are collected, and the business source logs are cleaned to remove irrelevant information in the business source logs; and different formats of the cleaned business source logs are converted into a unified format to obtain a target log; The target log is parsed to determine the system operation status of the service source to monitor the service source in real time.

8. A data synchronization device, characterized in that: Applied to a first data platform, the device comprises: a first acquisition unit, a second acquisition unit, an adjustment unit and a synchronization unit; wherein, The first acquisition unit is used to acquire change information of the service source and write the change information into the message middleware, wherein the change information is used to indicate the real-time service data generated in the service source; The second acquisition unit is used to acquire the data to be processed by the message middleware, where the data to be processed is the data formed after the message middleware performs business decoupling on the change information; The adjusting unit is used to adjust the data format of the to-be-processed data to a standardized data format to obtain target data; The synchronization unit is used to send the target data to the second data platform to achieve data synchronization. The second data platform refers to a data system for storing data in batches.

9. An electronic device, characterized in that: The method comprises a processor and a memory, wherein the memory is used to store one or more programs and is configured to be executed by the processor, wherein the programs include instructions for executing the steps in the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: A computer program for electronic data exchange is stored, wherein the computer program enables a computer to execute the method according to any one of claims 1 to 7.

Images