Data synchronization method, device, computer equipment and storage medium

Abstract

The present application is applicable to the field of data processing technology, and provides a data synchronization method, apparatus, computer equipment and storage medium, wherein the method includes: sending a data primary key query instruction to a target database, obtaining primary key information corresponding to the target data record fed back by the target database; generating a data query statement based on the primary key information, sending the data query statement to a source database, obtaining the target data output by the source database, and writing the target data to the target database, thereby reducing the occupancy of the data processing performance of the source database and not affecting the data query response speed of the source database.

Description

Technical Field

[0001] The present application belongs to the field of data processing technology, and in particular relates to a data synchronization method, apparatus, computer equipment and storage medium. Background Art

[0002] In recent years, intelligent investment research has used technological means to assist financial investment research and decision-making. Due to its advantages such as reducing transaction costs, improving investment research efficiency, enhancing analysis accuracy, and eliminating hidden risks, it has quickly gained favor among financial institutions. It is playing an increasingly important role in the asset management field and can greatly enhance the effectiveness and value of massive raw data in the financial market.

[0003] Currently, intelligent investment research applications require synchronizing large amounts of raw data stored in source databases to downstream databases for data analysis and utilization. Currently, the primary source database is an Oracle heterogeneous database. One solution for data synchronization based on Oracle heterogeneous databases is OGG (Oracle Golden Gate). This solution primarily involves the source database parsing Oracle database archive logs, extracting each data change as a change event, and writing this change event to a message queue. The downstream database then parses each data change event, converting it into insert, update, and delete operations within the downstream database, synchronously updating the downstream database tables.

[0004] The advantage of this solution is that it can meet the real-time requirements of data synchronization. However, its disadvantage is that the source database needs to frequently extract data change events, which occupies a large amount of the source database's data processing performance and reduces the source database's data query response speed. Summary of the Invention

[0005] Embodiments of the present application provide a data synchronization method, apparatus, computer device, and storage medium to address the problem in the prior art that data synchronization methods between databases occupy a large amount of data processing performance of the source database and reduce the data query response speed of the source database.

[0006] A first aspect of an embodiment of the present application provides a data synchronization method, including:

[0007] Sending a data primary key query instruction to the target database, wherein the data primary key query instruction is used to instruct the target database to search for primary key information corresponding to the target data record;

[0008] Obtaining primary key information corresponding to the target data record fed back by the target database;

[0009] Generate a data query statement based on the primary key information, and send the data query statement to a source database, wherein the data query statement is used to instruct the source database to search for target data having a storage time correlation with the primary key information, wherein the storage time correlation includes a storage time that is later than a storage time of the target data record corresponding to the primary key information;

[0010] The target data output by the source database is obtained, and the target data is written into the target database.

[0011] A second aspect of an embodiment of the present application provides a data synchronization device, including:

[0012] A first sending module is used to send a data primary key query instruction to a target database, wherein the data primary key query instruction is used to instruct the target database to search for primary key information corresponding to a target data record;

[0013] An acquisition module, configured to acquire primary key information corresponding to the target data record fed back by the target database;

[0014] a second sending module, configured to generate a data query statement based on the primary key information, and send the data query statement to a source database, wherein the data query statement is configured to instruct the source database to search for target data having a storage time correlation with the primary key information, wherein the storage time correlation includes a storage time that is later than a storage time of the target data record corresponding to the primary key information;

[0015] The data writing module is used to obtain the target data output by the source database and write the target data into the target database.

[0016] A third aspect of an embodiment of the present application provides a terminal, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the method described in the first aspect when executing the computer program.

[0017] A fourth aspect of an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the method described in the first aspect are implemented.

[0018] A fifth aspect of the present application provides a computer program product, which, when executed on a terminal, enables the terminal to execute the steps of the method described in the first aspect.

[0019] As can be seen from the above, in this embodiment, the synchronization data required in the data synchronization process is obtained by searching and reading data in the source database, and then the relevant data is written to the target database, and data transfer is realized between the target database and the source database. While improving the data synchronization frequency and the real-time performance of data synchronization, it reduces the occupancy of the data processing performance of the source database and does not affect the data query response speed of the source database. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the embodiments or descriptions of the prior art. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0021] Figure 1 This is a process of a data synchronization method provided by the embodiment of the present application Figure 1 ;

[0022] Figure 2 This is a process of a data synchronization method provided by the embodiment of the present application Figure 2 ;

[0023] Figure 3 This is a structural diagram of a data synchronization device provided in an embodiment of the present application;

[0024] Figure 4 This is a structural diagram of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0025] In the following description, specific details such as specific system structures and techniques are provided for purposes of illustration rather than limitation to facilitate a thorough understanding of the embodiments of the present application. However, it will be apparent to those skilled in the art that the present application may be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to avoid obscuring the description of the present application with unnecessary detail.

[0026] It will be understood that when used in this specification and the appended claims, the term "comprising" indicates the presence of described features, integers, steps, operations, elements and / or components, but does not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof.

[0027] It should also be understood that the terms used in this specification are for the purpose of describing specific embodiments only and are not intended to limit the present application. As used in this specification and the appended claims, the singular forms "a," "an," and "the" are intended to include the plural forms unless the context clearly indicates otherwise.

[0028] It should be further understood that the term "and / or" used in this specification and the appended claims refers to and includes any and all possible combinations of one or more of the associated listed items.

[0029] As used in this specification and the appended claims, the term "if" can be interpreted as "when" or "upon" or "in response to determining" or "in response to detecting," depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" can be interpreted as meaning "upon determination" or "in response to determining" or "upon detection of [described condition or event]" or "in response to detecting [described condition or event]," depending on the context.

[0030] In specific implementations, the terminals described in the embodiments of the present application include, but are not limited to, other portable devices such as mobile phones, laptop computers, or tablet computers with touch-sensitive surfaces (e.g., touch screen displays and / or touch pads). It should also be understood that in some embodiments, the device is not a portable communication device, but a desktop computer with a touch-sensitive surface (e.g., touch screen displays and / or touch pads).

[0031] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, and the like. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, and the like that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments in which tasks are performed by remote processing devices connected via a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media, including storage devices.

[0032] In the following discussion, a terminal including a display and a touch-sensitive surface is described. However, it should be understood that the terminal may include one or more other physical user interface devices such as a physical keyboard, mouse, and / or joystick.

[0033] The terminal supports various applications, such as one or more of the following: a drawing application, a presentation application, a word processing application, a website creation application, a disk burning application, a spreadsheet application, a game application, a phone application, a video conferencing application, an email application, an instant messaging application, a workout support application, a photo management application, a digital camera application, a digital video camera application, a web browsing application, a digital music player application, and / or a digital video player application.

[0034] Various applications that can be executed on the terminal can use at least one common physical user interface device, such as a touch-sensitive surface. One or more functions of the touch-sensitive surface and corresponding information displayed on the terminal can be adjusted and / or changed between applications and / or within a corresponding application. In this way, the common physical architecture of the terminal (e.g., the touch-sensitive surface) can support a variety of applications with user interfaces that are intuitive and transparent to the user.

[0035] It should be understood that the size of the serial numbers of each step in this embodiment does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of this application.

[0036] This solution is applied to data synchronization and migration scenarios. By setting up a data synchronization platform and using data query, data transfer is realized between the target database and the source database. While increasing the frequency and real-time performance of data synchronization, it reduces the occupancy of the source database's data processing performance without affecting the source database's data query response speed.

[0037] In order to illustrate the technical solution described in this application, specific embodiments are provided below.

[0038] See also Figure 1 , Figure 1 This is a process of a data synchronization method provided by the embodiment of the present application Figure 1 .like Figure 1 As shown, a data synchronization method includes the following steps:

[0039] Step 101, sending a data primary key query instruction to the target database;

[0040] Step 102: Obtain primary key information corresponding to the target data record fed back by the target database;

[0041] Step 103: Generate a data query statement based on the primary key information, and send the data query statement to the source database;

[0042] Step 104: Obtain the target data output by the source database, and write the target data into the target database.

[0043] The specific implementation process of each of the above steps will be described below.

[0044] Wherein, in step 101, a data primary key query instruction is sent to a target database, where the data primary key query instruction is used to instruct the target database to search for primary key information corresponding to a target data record.

[0045] Optionally, the target database and the source database are part of a heterogeneous database system.

[0046] A heterogeneous database system is a collection of multiple related database systems that can achieve data sharing and transparent access. Several database systems already exist before joining the heterogeneous database system and have their own database management systems. Each component of the heterogeneous database has its own autonomy, and while achieving data sharing, each database system still has its own application characteristics, integrity control and security control.

[0047] In one specific application, the target database includes data tables partitioned according to a set time granularity, partition indexes are constructed for partitions of different data tables, and data in the data tables have primary key information assigned sequentially according to storage time.

[0048] Specifically, the data tables in the target database are partitioned by time, with the partition granularity being, for example, months. For example, 202001, 202002, 202003, etc., each partitioned data table stores the data for that month. A partition index is established for the data tables in the target database. When a data primary key query instruction is received from the data synchronization platform, the corresponding data partition can be quickly located based on the date, quickly filtering out irrelevant data.

[0049] Among them, each data record in the data table has a primary key, and the primary key can be assigned an incremental value according to the storage time relationship, so that the primary key information corresponding to the target data record can be quickly located.

[0050] Furthermore, the target data record that the data primary key query instruction is used to search is specifically the data record with the latest storage time in the target database, and the primary key information corresponding to the target data record has the latest storage time in the target database.

[0051] Sending the data primary key query instruction to the target database may be, when a synchronization instruction for target data from the client is obtained, sending the data primary key query instruction to the target database based on the synchronization instruction.

[0052] Create a search-like financial artificial intelligence investment research engine on the client side - through a simple search engine format, it provides functions such as event causal reasoning in the financial field, tracing the impact of macro events on investment target prices, accurate question-and-answer on investment issues, and rapid aggregation of mainstream research views, to assist in improving the efficiency of investment research on major asset classes.

[0053] While satisfying the user's search-based investment research engine, data synchronization is achieved in a timely manner.

[0054] The target server and subsequent source servers can be independent servers or cloud servers that provide basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms.

[0055] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Artificial Intelligence (AI) is the theory, method, technology, and application system that uses digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use knowledge to achieve optimal results.

[0056] Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interaction systems, and mechatronics. AI software technologies primarily encompass computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning.

[0057] Step 102: Obtain primary key information corresponding to the target data record fed back by the target database.

[0058] The target database matches the target partition locally according to the data primary key query instruction, and searches the target partition for the primary key information corresponding to the target data record. Specifically, it can be the target primary key of the specified date or the target primary key with the latest storage time.

[0059] The target primary key is obtained to indicate which data node in the source database to start data synchronization from in the subsequent data synchronization process.

[0060] In one embodiment, the target database includes a data table partitioned according to a set time granularity, partition indexes are constructed for different partitions of the data table, and when the data in the data table has primary key information assigned sequentially according to storage time, step 102 of obtaining primary key information corresponding to target data records fed back by the target database includes:

[0061] Obtain the target database based on the data primary key query instruction, and match the target data records from the data table according to the partition index of the data table and the storage time corresponding to the primary key information;

[0062] Extract the primary key information corresponding to the target data record.

[0063] When the target database receives the data primary key query instruction from the data synchronization platform, the database can quickly locate the corresponding data partition based on the key information in the data primary key query instruction, such as the information storage date, keywords in the information, etc., filter out irrelevant data, and further quickly locate the target data record. The data synchronization platform can obtain the corresponding primary key information based on the target data record, thereby improving data processing efficiency.

[0064] Step 103: Generate a data query statement based on the primary key information, and send the data query statement to the source database.

[0065] This data query statement is used to instruct the source database to search for target data that has a storage time association with primary key information.

[0066] Specifically, the source database refers to a database that directly provides original information or specific data. In the embodiment of the present application, the source database is specifically a source database of target data in the process of synchronizing target data to the target database.

[0067] The storage time association includes that the storage time is after the storage time of the target data record corresponding to the primary key information, that is, the data query statement is used to instruct the source database to search for target data with a time association with the primary key information, the storage time of which is after the storage time of the target data record corresponding to the primary key information.

[0068] Alternatively, a specific time association can be set as needed, such as synchronizing historical data whose storage time is within a set period before the storage time of the target data record corresponding to the primary key information to achieve update synchronization of historical data. No specific limitation is made here.

[0069] Here, when data is synchronized between the target database and the source database through the data synchronization platform, the source database obtains the target data to be synchronized through data query. In the embodiment of the present application, considering the performance impact on the source database master database, data synchronization is based on a read-only operation on the source database.

[0070] Specifically, when reading data from the source database, the data can be read from the backup database of the source database. The data in the backup database is consistent with the data in the main database of the source database, which reduces the data processing burden of the source data, reduces the data processing resource usage of the source data, and improves the response speed and processing speed of data synchronization.

[0071] In one specific application, the source database includes data tables partitioned according to a set time granularity, partition indexes are constructed for partitions of different data tables, and the data in the data tables have primary key information assigned sequentially according to storage time.

[0072] The source database includes database table partitions. The database table in each partition includes a number of stored data. The primary keys in the stored data are arranged and assigned values ​​in chronological order.

[0073] The data tables in the source database can be partitioned by time, with the partition granularity being, for example, months. For example, 202001, 202002, and 202003. Each partitioned data table stores the data for that month. A partitioned index is established for the data tables in the target database. When receiving a data primary key query command from the data synchronization platform, the corresponding data partition can be quickly located based on the date, quickly filtering out irrelevant data.

[0074] Among them, each data record in the data table has a primary key, and the primary key can be assigned incrementally according to the storage time relationship, so as to quickly locate the primary key information corresponding to the target data record, and at the same time support the fast page turning query function.

[0075] Furthermore, when generating a data query statement based on primary key information, the data storage time corresponding to the primary key information is parsed based on the primary key information, and based on the data storage time, a SQL query statement is assembled and submitted to the source database. Alternatively, the SQL query statement may be assembled based on the data storage time, combined with the data table naming rules and data storage rules in the source database, and the number of page-turning queries indicated, and submitted to the source database. This is not specifically limited here, and the query statement can be generated based on actual data query requirements and the size of currently available data processing resources.

[0076] In addition, it should be noted that in an application scenario of this embodiment, the source database and the target database can be databases located in different subsidiaries under the same group company. In this embodiment, in order to achieve data synchronization across subsidiaries, a data synchronization platform is added between the two databases as a data transfer. The data queried from the source database is written to the target database through the data synchronization platform server in the form of data query. With the help of the data synchronization platform, the point-to-point data synchronization between databases in the existing technology can be avoided, ensuring data security and the controllability and reliability of data synchronization operations.

[0077] Therefore, in a specific implementation process, step 103 may be to output the acquired primary key information to a cross-company data server, generate a data query statement based on the primary key information through the data server, and send the data query statement to a source database located in the same subsidiary.

[0078] Step 104: Obtain the target data output by the source database, and write the target data into the target database.

[0079] In one embodiment, when the source database includes a data table partitioned according to a set time granularity, partition indexes are constructed for different partitions of the data table, and the data in the data table has primary key information assigned sequentially according to storage time, step 104 obtains target data output by the source database and writes the target data into the target database, including:

[0080] Obtain target data output by the source database after performing a page-turning query from the data table based on the data query statement and according to the storage time corresponding to the partition index and primary key information of the data table; wherein the target data includes the data query results corresponding to each page-turning query output by the source database;

[0081] Each time a data query result is obtained, it is written to the target database until all target data is written.

[0082] In the database field, time granularity refers to the degree of detail in data processing under the time dimension. In this embodiment, time granularity specifically refers to the time length used for each data partition when partitioning a data table.

[0083] For example, if one month is set as the time granularity, when partitioning the data table according to the time granularity, the length of one month is used as the unit, and the data tables generated in the same month are divided into the same partition.

[0084] After the source database obtains the data query statement, the data page query method is specifically adopted. For each page of the data table in the database queried, the corresponding page of query data is returned for data feedback. Each time the data synchronization platform obtains a page of data query results, it writes a page of data to the target database, thereby improving the synchronicity and timeliness of data synchronization.

[0085] As an optional implementation, the data synchronization method in the embodiment of the present application further includes:

[0086] When data query instructions are obtained from different clients, a query instruction queue is established based on the data query instructions;

[0087] Based on the query instruction queue, data query instructions are output to the target database in sequence;

[0088] When query data output by the target database based on the data query instruction is obtained, the query data is cached to obtain a data cache list;

[0089] Match the query data in the data cache list with the data query instructions for key information, and obtain matching data corresponding to different data query instructions;

[0090] The matching data is output to the client that sent the data query instruction.

[0091] Among them, the data synchronization platform can also implement data query functions. When obtaining data query instructions from multiple consumer clients, the data synchronization platform can use the coroutine in the Kotlin language as the worker coroutine of the data synchronization platform. For the worker coroutine of data query, a queue is established, and a first-in-first-out method is used to query the target database in turn based on the data query instructions of the consumer client.

[0092] After obtaining the matching data fed back by the target database, a cache is created for the matching results, and data output matching is performed based on the data query instructions of each consumer client. If the date, keyword and other elements match, it is considered a cache hit. At this time, the data is directly taken from the cache and fed back to the user, improving the user query response speed under multiple data query conditions.

[0093] In the existing technology, since the source database needs to ensure timely response to local data processing operations, it also needs to perform data synchronization operations with other databases, which consumes server resources. Usually, a data operation queue is formed based on the server log in the source database. When achieving data synchronization, it is also necessary to access database synchronization tools, synchronization components, etc., which requires significant changes to the source database.

[0094] In this embodiment, the synchronization data required in the data synchronization process is obtained by searching and reading data in the source database, and then the relevant data is written to the target database, and data transfer is realized between the target database and the source database. While improving the data synchronization frequency and the real-time performance of data synchronization, it reduces the occupancy of the data processing performance of the source database and does not affect the data query response speed of the source database.

[0095] The embodiments of the present application also provide different implementation methods of the data synchronization method.

[0096] See also Figure 2 , Figure 2 This is a process of a data synchronization method provided by the embodiment of the present application Figure 2 .like Figure 2 As shown, a data synchronization method includes the following steps:

[0097] Step 201: Acquire data synchronization indication information.

[0098] The data synchronization indication information includes existing data synchronization information and incremental data synchronization information.

[0099] Here, data synchronization targets are divided into two types: existing data and incremental data. The synchronization process for existing data requires less real-time synchronization, and the volume of existing data is relatively large. Therefore, a different data synchronization processing method is used for this purpose than for incremental data.

[0100] Step 202: Based on the data synchronization instruction information and in combination with the incremental data synchronization information, a data primary key query instruction is generated to instruct the target database to search for primary key information corresponding to the incremental data record with the latest storage date.

[0101] Based on the incremental data synchronization information, it is determined that the incremental data synchronization operation needs to be performed. For the synchronization of incremental data, it is usually necessary to find the current latest data storage time from the target database. Based on the latest data, the data in the source database that is after the latest storage time is used as the target data for data synchronization.

[0102] When generating a data primary key query instruction based on the data synchronization indication information and in combination with the incremental data synchronization information to instruct the target database to search for the primary key information corresponding to the incremental data record with the latest storage date, specifically based on the data synchronization indication information, parsing the data information indicated by the incremental data synchronization information, such as storage time, data keywords, etc., assembling them into a SQL query statement, and obtaining the data primary key query instruction.

[0103] In the embodiment of the present application, steps 202 to 206 implement data synchronization processing operations for incremental data.

[0104] Step 203: Send a data primary key query instruction to the target database.

[0105] The data primary key query instruction is used to instruct the target database to search for primary key information corresponding to the target data record.

[0106] Step 204: Obtain primary key information corresponding to the target data record fed back by the target database.

[0107] The implementation process of this step is the same as the implementation process of step 102 in the aforementioned embodiment, and will not be repeated here.

[0108] Step 205: Generate a data query statement based on the primary key information, and send the data query statement to the source database.

[0109] The data query statement is used to instruct the source database to search for target data having a storage time correlation with the primary key information, where the storage time correlation includes a storage time that is later than a storage time of the target data record corresponding to the primary key information.

[0110] The implementation process of this step is the same as the implementation process of step 103 in the aforementioned embodiment, and will not be repeated here.

[0111] Step 206: Obtain the target data output by the source database, and write the target data into the target database.

[0112] The implementation process of this step is the same as the implementation process of step 104 in the aforementioned embodiment, and will not be repeated here.

[0113] The above steps mainly realize the real-time synchronization of incremental data in the database. The existing data in the database needs to be processed through the big data platform.

[0114] In the embodiment of the present application, steps 207 to 208 implement data synchronization processing operations for incremental data.

[0115] Step 207: Determine the stock data to be synchronized in the source database based on the stock data synchronization information included in the data synchronization indication information.

[0116] The existing data synchronization information may include the data storage start time and end time, number of data items, data content, and other information of the existing data to be synchronized, so as to determine the existing data to be synchronized in the source database based on the existing data synchronization information.

[0117] Step 208: Import the existing data into the big data platform to form a transit file in a set format, and export the transit file to the target database based on the big data platform.

[0118] The big data platform can be a cloud data platform to achieve timely processing of large amounts of data.

[0119] To import existing data into a big data platform and create a transit file in a predefined format, you can use the data migration tool Sqoop as a data synchronization tool. Sqoop is a tool for transferring data between Hadoop and relational databases. It can import data from a relational database (such as MySQL, Oracle, Postgres, etc.) into Hadoop's HDFS, and it can also import data from HDFS into a relational database.

[0120] For each day's target inventory data in each table in the source database, use the Sqoop tool to import it into the big data platform to form an HDFS file in the Hadoop distributed file system. Then, use the Spark tool configured on the big data platform to export the HDFS file to the target database.

[0121] Different data synchronization processing methods are used for incremental data and existing data, and incremental data is processed separately from large amounts of existing data to improve the real-time synchronization of incremental data and reduce data synchronization delays.

[0122] In this embodiment, the objects of data synchronization are divided into two types: existing data and incremental data. Different data synchronization processing methods are used for incremental data and existing data. Incremental data is processed separately from large amounts of existing data, thereby improving the real-time synchronization of incremental data and reducing data synchronization delays. While increasing the data synchronization frequency and the real-time performance of data synchronization, it reduces the occupancy of the data processing performance of the source database and does not affect the data query response speed of the source database.

[0123] See also Figure 3 , Figure 3 This is a structural diagram of a data synchronization device provided in an embodiment of the present application. For the sake of convenience, only the parts related to the embodiment of the present application are shown.

[0124] The data synchronization device 300 includes:

[0125] A first sending module 301 is configured to send a data primary key query instruction to a target database, wherein the data primary key query instruction is configured to instruct the target database to search for primary key information corresponding to a target data record;

[0126] An acquisition module 302 is configured to acquire primary key information corresponding to the target data record fed back by the target database;

[0127] A second sending module 303 is configured to generate a data query statement based on the primary key information and send the data query statement to a source database, wherein the data query statement is configured to instruct the source database to search for target data having a storage time correlation with the primary key information, where the storage time correlation includes a storage time that is later than a storage time of the target data record corresponding to the primary key information;

[0128] The data writing module 304 is configured to obtain the target data output by the source database and write the target data into the target database.

[0129] The target database includes a data table partitioned according to a set time granularity, different partitions of the data table are constructed with partition indexes, and the data in the data table has primary key information assigned in sequence according to the storage time;

[0130] The acquisition module is specifically configured to:

[0131] Obtaining the target data record matched from the data table based on the data primary key query instruction and according to the partition index of the data table and the storage time corresponding to the primary key information;

[0132] Extract the primary key information corresponding to the target data record.

[0133] The first sending module is specifically configured to:

[0134] Acquire data synchronization indication information, wherein the data synchronization indication information includes existing data synchronization information and incremental data synchronization information;

[0135] Based on the data synchronization instruction information and in combination with the incremental data synchronization information, a data primary key query instruction is generated for instructing the target database to search for primary key information corresponding to the incremental data record with the latest storage date;

[0136] Send the data primary key query instruction to the target database.

[0137] The device also includes:

[0138] The existing data synchronization module is used to:

[0139] Determining the stock data to be synchronized in the source database based on the stock data synchronization information included in the data synchronization indication information;

[0140] The stock data is imported into a big data platform to form a transfer file in a set format, and the transfer file is exported to the target database based on the big data platform.

[0141] The source database includes data tables partitioned according to a set time granularity, different partitions of the data tables are constructed with partition indexes, and the data in the data tables have primary key information assigned in sequence according to storage time.

[0142] The data writing module is specifically used to:

[0143] Obtaining the target data output by the source database after performing a page-turning query from the data table based on the data query statement and according to the storage time corresponding to the partition index and primary key information of the data table, wherein the target data includes data query results corresponding to each page-turning query output by the source database;

[0144] Each time the data query result is obtained, the data query result is written into the target database until all the target data are written.

[0145] The device also includes:

[0146] Data query module, used to:

[0147] When data query instructions from different clients are obtained, a query instruction queue is established based on the data query instructions;

[0148] Based on the query instruction queue, outputting the data query instructions to the target database in sequence;

[0149] When query data output by the target database based on the data query instruction is obtained, the query data is cached to obtain a data cache list;

[0150] Matching the query data in the data cache list with the data query instructions respectively for key information, and obtaining matching data corresponding to different data query instructions;

[0151] The matching data is outputted to the client that sent the data query instruction.

[0152] The data synchronization device provided in the embodiment of the present application can implement each process of the embodiment of the above-mentioned data synchronization method and can achieve the same technical effect. To avoid repetition, it will not be described here.

[0153] Figure 4 4 is a structural diagram of a computer device provided in an embodiment of the present application. As shown in the figure, the computer device 4 of this embodiment includes: at least one processor 40 ( Figure 4Only one is shown), a memory 41 and a computer program 42 stored in the memory 41 and executable on the at least one processor 40, wherein the processor 40 implements the steps of any of the above-mentioned method embodiments when executing the computer program 42.

[0154] The computer device 4 can be a desktop computer, a notebook computer, a PDA, a cloud server or other computing devices. The computer device 4 can include, but is not limited to, a processor 40 and a memory 41. Those skilled in the art will understand that Figure 4 It is only an example of the computer device 4 and does not constitute a limitation of the computer device 4. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, the computer device may also include input and output devices, network access devices, buses, etc.

[0155] The processor 40 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor.

[0156] The memory 41 may be an internal storage unit of the computer device 4, such as a hard disk or memory of the computer device 4. The memory 41 may also be an external storage device of the computer device 4, such as a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) card, a Flash Card, etc. equipped on the computer device 4. Furthermore, the memory 41 may include both an internal storage unit of the computer device 4 and an external storage device. The memory 41 is used to store the computer program and other programs and data required by the computer device. The memory 41 may also be used to temporarily store data that has been output or is about to be output.

[0157] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated into one processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0158] In the above embodiments, the description of each embodiment has its own focus. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant description of other embodiments.

[0159] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0160] In the embodiments provided in this application, it should be understood that the disclosed devices / terminals and methods can be implemented in other ways. For example, the device / terminal embodiments described above are merely illustrative. For example, the division of the modules or units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0161] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0162] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0163] If the integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the process in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and when the computer program is executed by the processor, it can implement the steps of the above-mentioned various method embodiments. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium. It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.

[0164] The present application implements all or part of the processes in the above-mentioned embodiment method, and can also be implemented through a computer program product. When the computer program product runs on a terminal, the terminal can implement the steps in the above-mentioned method embodiments when executing the computer program product.

[0165] The above-described embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present application, and should all be included in the scope of protection of the present application.

Claims

1. A data synchronization method, characterized in that: include: Sending a data primary key query instruction to the target database, wherein the data primary key query instruction is used to instruct the target database to search for primary key information corresponding to the target data record; The data primary key query instruction is used to instruct the target database to search for primary key information corresponding to the incremental data record with the latest storage date; Obtaining target data records matched from a data table included in the target database based on the data primary key query instruction and according to the partition index of the data table and the storage time corresponding to the primary key information; Extracting primary key information corresponding to the target data record; Generate a data query statement based on the primary key information, send the data query statement to the source database, and obtain the target data to be synchronized from the source database in a data query manner; The data query statement is used to instruct the source database to search for the target data having a storage time correlation with the primary key information, wherein the storage time correlation includes that the storage time is after the storage time of the target data record corresponding to the primary key information; The target data output by the source database is obtained, and the target data is written into the target database.

2. The method according to claim 1, characterized in that The target database includes the data table partitioned according to a set time granularity, different partitions of the data table are constructed with partition indexes, and the data in the data table has primary key information assigned in sequence according to storage time.

3. The method according to claim 1, characterized in that The sending of a data primary key query instruction to the target database includes: Acquire data synchronization indication information, wherein the data synchronization indication information includes existing data synchronization information and incremental data synchronization information; Based on the data synchronization instruction information and in combination with the incremental data synchronization information, a data primary key query instruction is generated for instructing the target database to search for primary key information corresponding to the incremental data record with the latest storage date; Send the data primary key query instruction to the target database.

4. The method according to claim 3, characterized in that After obtaining the data synchronization indication information, the method further includes: Determining the stock data to be synchronized in the source database based on the stock data synchronization information included in the data synchronization indication information; The stock data is imported into a big data platform to form a transfer file in a set format, and the transfer file is exported to the target database based on the big data platform.

5. The method according to claim 1, characterized in that The source database includes a data table partitioned according to a set time granularity, different partitions of the data table are constructed with partition indexes, and the data in the data table has primary key information assigned in sequence according to storage time.

6. The method according to claim 5, characterized in that The acquiring the target data output by the source database and writing the target data into the target database includes: Obtaining the target data output by the source database after performing a page-turning query from the data table based on the data query statement and according to the storage time corresponding to the partition index and primary key information of the data table, wherein the target data includes data query results corresponding to each page-turning query output by the source database; Each time the data query result is obtained, the data query result is written into the target database until all the target data are written.

7. The method according to claim 1, characterized in that Also includes: When data query instructions from different clients are obtained, a query instruction queue is established based on the data query instructions; Based on the query instruction queue, outputting the data query instructions to the target database in sequence; When query data output by the target database based on the data query instruction is obtained, the query data is cached to obtain a data cache list; Matching the query data in the data cache list with the data query instructions respectively for key information, and obtaining matching data corresponding to different data query instructions; The matching data is outputted to the client that sent the data query instruction.

8. A data synchronization device, characterized in that: include: A first sending module is configured to send a data primary key query instruction to a target database, wherein the data primary key query instruction is configured to instruct the target database to search for primary key information corresponding to a target data record; the data primary key query instruction is configured to instruct the target database to search for primary key information corresponding to an incremental data record having the latest storage date; An acquisition module is configured to acquire, based on the data primary key query instruction, a target data record matched from a data table included in the target database according to the partition index of the data table and the storage time corresponding to the primary key information; and extract the primary key information corresponding to the target data record; A second sending module is configured to generate a data query statement based on the primary key information, send the data query statement to the source database, and obtain the target data to be synchronized from the source database in a data query manner; The data query statement is used to instruct the source database to search for the target data having a storage time correlation with the primary key information, wherein the storage time correlation includes that the storage time is after the storage time of the target data record corresponding to the primary key information; The data writing module is used to obtain the target data output by the source database and write the target data into the target database.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images