A real-time synchronous backup method for financial data and related equipment

Abstract

The present application provides a method and related equipment for real-time synchronous backup of financial data, which determines the log playback position of high-frequency financial information in the process of data synchronous backup according to historical log records; determines the data inclination through the log playback position and the basic synchronization record, and then determines the data freshness according to the data inclination and the transaction synchronization record; performs coupling detection on the conflict between data reading and data synchronization through each data reading record and the log playback position, and obtains the conflict coupling entropy between data reading and data synchronization of the corresponding data node; when the high-frequency financial information in the target financial data is synchronously backed up, each data node of the high-frequency financial information is synchronously backed up in parallel based on the data freshness and each conflict coupling entropy. Based on the above scheme, the parallel synchronous backup of the data nodes in the financial data can be realized by means of log parallel playback, thereby improving the data freshness of the financial data in the database.

Description

Technical Field

[0001] The present application relates to the technical field of data synchronization and backup, and more specifically, to a financial data real-time synchronization and backup method and related equipment. Background Art

[0002] Real-time synchronous backup of financial data refers to the process of copying data from a primary database to one or more backup databases in a financial management system in real time or periodically to ensure data security, consistency, and high availability. Data synchronous backup is particularly important in modern financial systems because financial data is of high value and high sensitivity. Any data loss or damage may lead to serious business interruption or financial loss.

[0003] In the prior art, real-time synchronous backup of financial data usually uses database replication, log transfer and data streaming to synchronize data changes from the main database to the backup database in real time. Common methods include master-slave replication and multi-master replication. When data is unevenly distributed among data nodes, some data nodes will be overloaded and affect data synchronization performance, thereby failing to ensure the timeliness of synchronized data, resulting in a large difference between the data in the backup database and the latest state of the main database, that is, the data freshness is not high enough. Parallel synchronous backup of data nodes in financial data can be achieved by replaying logs in parallel, so that the number of synchronization threads for each data point can be controlled according to the amount of synchronized data in each data node, thereby avoiding the problem of low data freshness caused by uneven load distribution. Therefore, how to perform parallel synchronous backup of data nodes in financial data to avoid uneven load distribution of different data nodes and improve the data freshness of financial data in the database is a difficult problem faced by the industry. Summary of the invention

[0004] The present application provides a real-time synchronous backup method for financial data and related equipment, which can realize parallel synchronous backup of data nodes in financial data by means of parallel playback of logs, thereby improving the data freshness of financial data in the database.

[0005] In a first aspect, the present application provides a method for real-time synchronization and backup of financial data, comprising:

[0006] Monitor the log information of the target financial data in the database, and then obtain the historical log records of the target financial data;

[0007] Filtering out high-frequency financial information from the target financial data through the historical log records, and then determining the log playback position of the high-frequency financial information in the data synchronization backup process according to the historical log records;

[0008] Dividing the historical log records into transaction synchronization records and basic synchronization records, determining the data inclination of the high-frequency financial information during log playback through the log playback site and the basic synchronization records, and then determining the data freshness of the high-frequency financial information according to the data inclination and the transaction synchronization records;

[0009] Obtaining data reading records of the high-frequency financial information in each data node, performing coupling detection on the conflict between data reading and data synchronization through each data reading record and the log playback location, and obtaining the conflict coupling entropy between data reading and data synchronization of the corresponding data node;

[0010] When the high-frequency financial information in the target financial data is synchronously backed up, each data node of the high-frequency financial information is synchronously backed up in parallel based on the data freshness and each conflict coupling entropy.

[0011] In some embodiments, determining the log playback location of the high-frequency financial information in the data synchronization backup process according to the historical log record specifically includes:

[0012] For each high-frequency data table in the high-frequency financial information, obtaining the to-be-synchronized log record of the high-frequency data table in the historical log record;

[0013] Determine the playback queue of the high-frequency data table at the current moment through the log record to be synchronized, and then obtain the playback queue of each high-frequency data table at the current moment;

[0014] The log playback position of the high-frequency financial information in the data synchronization backup process is determined according to the log position of the initial playback data corresponding to each playback queue.

[0015] In some embodiments, determining the data inclination of the high-frequency financial information during log playback through the log playback location and the basic synchronization record specifically includes:

[0016] Extracting the operation synchronization queue of each high-frequency data table in the high-frequency financial information at the current moment in the basic synchronization record through the log playback site;

[0017] Determine the data synchronization deviation of the corresponding high-frequency data table according to each operation synchronization queue;

[0018] The data skewness of the high-frequency financial information during log playback is determined based on all data synchronization deviations.

[0019] In some embodiments, determining the data freshness of the high-frequency financial information according to the data inclination and the transaction synchronization record specifically includes:

[0020] Obtaining all transaction logs in the transaction synchronization record;

[0021] Determine the log sending time of each transaction log during the data synchronization process through the transaction synchronization record;

[0022] Counting the transaction synchronization records to obtain a synchronization transaction volume;

[0023] Determine the log playback time of the transaction operation in the transaction log according to the synchronization transaction volume and the data inclination;

[0024] The data freshness of the high-frequency financial information is determined according to the sending time of each log and the playback time of the log.

[0025] In some embodiments, dividing the historical log records into transaction synchronization records and basic synchronization records specifically includes:

[0026] Obtaining an operation type identifier for each log message in the historical log record;

[0027] Obtaining synchronized log records in the historical log records;

[0028] The synchronized log records are mapped and divided according to each operation type identifier to obtain transaction synchronization records and basic synchronization records.

[0029] In some embodiments, performing parallel synchronization backup on each data node of the high-frequency financial information based on the data freshness and each conflict coupling entropy specifically includes:

[0030] For each data node, obtain the conflict coupling entropy corresponding to the data node;

[0031] Determine the synchronization urgency of high-frequency financial information in the data node through the conflict coupling entropy and the data freshness;

[0032] If the synchronization urgency is greater than a preset urgency threshold, the high-frequency financial information in the data node is synchronously backed up, thereby completing the parallel synchronous backup of each data node.

[0033] In some embodiments, the database is a distributed database.

[0034] In a second aspect, the present application provides a real-time synchronous backup system for financial data, including a synchronous backup unit, wherein the synchronous backup unit includes:

[0035] An acquisition module is used to monitor the log information of the target financial data in the database, and then obtain the historical log records of the target financial data;

[0036] A processing module, configured to filter out high-frequency financial information from the target financial data through the historical log records, and then determine a log playback position of the high-frequency financial information in the data synchronization backup process according to the historical log records;

[0037] The processing module is also used to divide the historical log records into transaction synchronization records and basic synchronization records, determine the data inclination of the high-frequency financial information during log playback through the log playback site and the basic synchronization record, and then determine the data freshness of the high-frequency financial information according to the data inclination and the transaction synchronization record;

[0038] The processing module is further used to obtain the data reading record of the high-frequency financial information in each data node, and perform coupling detection on the conflict between data reading and data synchronization through each data reading record and the log playback position, so as to obtain the conflict coupling entropy between data reading and data synchronization of the corresponding data node;

[0039] The execution module is used to perform parallel synchronization backup on each data node of the high-frequency financial information based on the data freshness and each conflict coupling entropy when the high-frequency financial information in the target financial data is synchronized and backed up.

[0040] In a third aspect, the present application provides a computer device, comprising a memory and a processor, wherein the memory is used to store a computer program, and the processor is used to call and run the computer program from the memory, so that the computer device executes the above-mentioned real-time synchronization and backup method of financial data.

[0041] In a fourth aspect, the present application provides a computer-readable storage medium, in which instructions or codes are stored. When the instructions or codes are run on a computer, the computer implements the above-mentioned real-time synchronization and backup method of financial data when executing.

[0042] The technical solution provided by the embodiments disclosed in this application has the following beneficial effects:

[0043] In a real-time synchronous backup method for financial data and related equipment provided by the present application, log information of target financial data in a database is monitored to obtain historical log records of the target financial data; high-frequency financial information is screened out from the target financial data through the historical log records, and then the log playback position of the high-frequency financial information in the data synchronous backup process is determined according to the historical log records; the historical log records are divided into transaction synchronization records and basic synchronization records, and the data inclination of the high-frequency financial information during log playback is determined through the log playback position and the basic synchronization record, and then the data freshness of the high-frequency financial information is determined according to the data inclination and the transaction synchronization record; the data reading record of the high-frequency financial information in each data node is obtained, and the conflict between data reading and data synchronization is coupled detected through each data reading record and the log playback position to obtain the conflict coupling entropy between data reading and data synchronization of the corresponding data node; when the high-frequency financial information in the target financial data is synchronously backed up, each data node of the high-frequency financial information is synchronously backed up in parallel based on the data freshness and each conflict coupling entropy.

[0044] It can be seen that in this application, when the high-frequency financial information in the target financial data is synchronously backed up, each data node of the high-frequency financial information is synchronously backed up in parallel based on the data freshness and each conflict coupling entropy; wherein, the data freshness of the high-frequency financial information can be determined to obtain the real-time performance and delay of the high-frequency financial information during the synchronization process, and the above steps comprehensively consider the time it takes for the basic log and the transaction log to be sent from the main database to the backup server and the time it takes for data synchronization to be performed in the backup server, so that a more accurate delay of the high-frequency financial information during the synchronization process can be obtained, which is convenient for the subsequent parallel synchronous backup of the data nodes of the high-frequency financial information in the target financial data, thereby ensuring the data freshness of the financial data in the backup database, and then determining the conflict coupling entropy to obtain The quantitative value of the degree of conflict between the data reading process and the data synchronization process at the time of data use. The above steps can obtain a more accurate quantitative value of the degree of conflict between the data reading process and the data synchronization process by coupling detection of the conflict between the data reading process and the data synchronization process after the log playback point of each data node (that is, repeated time points), so as to facilitate the subsequent comprehensive consideration of the time conflict of data reading and data synchronization when backing up the high-frequency financial information in the target financial data, thereby reducing the delay of data backup and further improving the data freshness of the financial data in the database; in summary, based on the above scheme, the parallel synchronous backup of the data nodes in the financial data can be realized by means of parallel log playback, thereby improving the data freshness of the financial data in the database. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0046] Figure 1 is an exemplary flow chart of a method for real-time synchronization and backup of financial data according to some embodiments of the present application;

[0047] Figure 2 is a schematic diagram of a process for determining data tilt according to some embodiments of the present application;

[0048] Figure 3 It is a schematic diagram of the process of parallel synchronous backup shown in some embodiments of the present application;

[0049] Figure 4 is a schematic diagram of exemplary hardware and / or software of a synchronous backup unit according to some embodiments of the present application;

[0050] Figure 5 It is a structural diagram of a computer device for implementing a method for real-time synchronization and backup of financial data according to some embodiments of the present application. DETAILED DESCRIPTION

[0051] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0052] The embodiment of the present application provides a method for real-time synchronous backup of financial data and related equipment, the core of which is to determine the log playback position of high-frequency financial information in the data synchronization backup process according to historical log records; determine the data inclination through the log playback position and the basic synchronization record, and then determine the data freshness according to the data inclination and the transaction synchronization record; perform coupling detection on the conflict between data reading and data synchronization through each data reading record and the log playback position, and obtain the conflict coupling entropy between data reading and data synchronization of the corresponding data node; when the high-frequency financial information in the target financial data is synchronously backed up, each data node of the high-frequency financial information is synchronously backed up in parallel based on the data freshness and each conflict coupling entropy. Based on the above scheme, the parallel synchronous backup of the data nodes in the financial data can be realized by means of log parallel playback, thereby improving the data freshness of the financial data in the database.

[0053] In order to better understand the above technical solution, the above technical solution will be described in detail below in conjunction with the accompanying drawings and specific implementation methods. Figure 1 , which is an exemplary flow chart of a method for real-time synchronization and backup of financial data according to some embodiments of the present application, and the method for real-time synchronization and backup of financial data mainly includes the following steps:

[0054] In step 101, log information of target financial data in a database is monitored to obtain historical log records of the target financial data.

[0055] It should be noted that, in the present application, the database is a distributed database; the historical log record is the log information of the database operating on the target financial data; the target financial data refers to the financial data that needs to be processed, and the target financial data in the present application is a collection of data composed of multiple data tables; in specific implementation, the log information of the database operation on the target financial data within a preset time period (for example, the default is 1 week) is obtained in the database, and the collection of all log information can be used as a historical log record, wherein the log information includes the operation process, operation time, operation type identifier and synchronization identifier. The operation type identifier in the present application is divided into a transaction class identifier and a basic class identifier, and the synchronization identifier is divided into synchronized and unsynchronized. The synchronization identifier is synchronized. The log information also includes a synchronization start time and a synchronization end time. The synchronization start time is the timestamp for starting log synchronization of the log in the primary database, and the synchronization end time is the timestamp for completing the synchronization of the log in the backup database.

[0056] In step 102, high-frequency financial information is screened out from the target financial data through the historical log records, and then the log playback position of the high-frequency financial information in the data synchronization backup process is determined according to the historical log records.

[0057] In some embodiments, filtering out high-frequency financial information from the target financial data through the historical log records may be achieved by the following steps:

[0058] For each data table in the target financial data, the log frequency of the statistical data table in the historical log record;

[0059] Compare the log frequency with a preset high-frequency threshold, and if the log frequency is greater than the high-frequency threshold, use the data table as a high-frequency data table, and then obtain all high-frequency data tables;

[0060] Identify high-frequency financial information through all high-frequency data tables.

[0061] It should be noted that, in the present application, high-frequency financial information is a collection of data tables with higher frequency of use; in specific implementation, first, for each data table in the target financial data, the total number of log information involving the data table in the historical log records can be counted as the log frequency of the data table; then, the log frequency is compared with the preset high-frequency threshold. If the log frequency is less than or equal to the high-frequency threshold, no operation is performed. If the log frequency is greater than the high-frequency threshold, the data table is regarded as a high-frequency data table. All high-frequency data tables can be obtained in the above manner, wherein the high-frequency threshold represents the judgment threshold for high frequency in financial data, and the high-frequency threshold can be preset through historical experience; finally, the collection of all high-frequency data tables can be regarded as high-frequency financial information.

[0062] In some embodiments, determining the log playback position of the high-frequency financial information in the data synchronization backup process according to the historical log records can be implemented by the following steps:

[0063] For each high-frequency data table in the high-frequency financial information, obtaining the to-be-synchronized log record of the high-frequency data table in the historical log record;

[0064] Determine the playback queue of the high-frequency data table at the current moment through the log record to be synchronized, and then obtain the playback queue of each high-frequency data table at the current moment;

[0065] The log playback position of the high-frequency financial information in the data synchronization backup process is determined according to the log position of the initial playback data corresponding to each playback queue.

[0066] It should be noted that, in the present application, the log playback site represents the site where log playback starts when data synchronization is performed on each high-frequency data table. There are multiple values ​​for the log playback site, and each value corresponds to the log site of a high-frequency data table. In specific implementation, first, for each high-frequency data table in the high-frequency financial information, the set of log information that is synchronized and marked as unsynchronized in the historical log records can be used as the log records to be synchronized; then, all the log information in the log records to be synchronized is sorted in reverse order of the operation time, and the sorting result can be used as the playback queue of the high-frequency data table at the current moment. The above method can be used to obtain the playback queue of each high-frequency data table at the current moment. A playback queue, wherein the playback queue represents a collection of log information used for data synchronization by a high-frequency data table using log playback; finally, the log position corresponding to the first log information in each playback queue is used as the log position corresponding to the initial playback data, and the collection of all log positions can be used as the value range of the log playback position of the high-frequency financial information in the data synchronization and backup process, wherein, in the present application, the log position is a parameter used to identify a specific position in the database log, so that the replication or synchronization process can accurately continue to read and process the log data from this position; the initial playback data refers to the collection of high-frequency financial information that can be synchronized by the playback queue.

[0067] In step 103, the historical log records are divided into transaction synchronization records and basic synchronization records, and the data skewness of the high-frequency financial information during log playback is determined through the log playback site and the basic synchronization records, and then the data freshness of the high-frequency financial information is determined based on the data skewness and the transaction synchronization records.

[0068] In some embodiments, dividing the historical log records into transaction synchronization records and basic synchronization records can be implemented by the following steps:

[0069] Obtaining an operation type identifier for each log message in the historical log record;

[0070] Obtaining synchronized log records in the historical log records;

[0071] The synchronized log records are mapped and divided according to each operation type identifier to obtain transaction synchronization records and basic synchronization records.

[0072] It should be noted that, in the present application, the transaction synchronization record represents the process of data synchronization for transaction operations in database operations, and the basic synchronization record represents the process of data synchronization for basic operations (for example, addition, deletion, and modification) in database operations; in specific implementation, first, the operation type identifier of each log information in the historical log record is obtained; then, the set of all log information with the synchronization identifier as synchronized in the historical log record can be used as the synchronized log record; finally, the set of all log information with the operation type identifier as the transaction class identifier in the synchronized log record can be used as the transaction synchronization record, and the set of all log information with the operation type identifier as the basic class identifier in the synchronized log record can be used as the basic synchronization record.

[0073] In some embodiments, the data inclination of the high-frequency financial information during log playback is determined by the log playback location and the basic synchronization record, referring to Figure 2 The figure is a schematic diagram of a process for determining data tilt in some embodiments of the present application. In this embodiment, determining data tilt can be achieved by using the following steps:

[0074] In step 1031, the operation synchronization queue for each high-frequency data table in the high-frequency financial information at the current moment is extracted from the basic synchronization record through the log playback site;

[0075] In step 1032, the data synchronization deviation of the corresponding high-frequency data table is determined according to each operation synchronization queue;

[0076] In step 1033, the data skewness of the high-frequency financial information during log playback is determined based on all data synchronization deviations.

[0077] It should be noted that, in the present application, data skew indicates the degree of imbalance in data loads of different data nodes during log playback. Excessive data skew may cause data node overload, thereby affecting data consistency in data synchronization. In specific implementation, first, for each high-frequency data table in the high-frequency financial information, a collection of all log information related to the high-frequency data table is obtained in the basic synchronization record as a data table log record, and the value of the log playback position corresponding to the high-frequency data table is obtained as the log position, and then all log information after the log position is obtained in the data table log record. All log information can be arranged in reverse chronological order as the operation synchronization queue of the high-frequency data table. The above method can be used to obtain the high-frequency financial information of each high-frequency data table. operation synchronization queue; then, for each high-frequency data table, obtain the operation synchronization queue corresponding to the high-frequency data table, and use the absolute value of the difference between the synchronization start time and the synchronization end time in each log information in the operation synchronization queue as the synchronization delay time of the corresponding log information. The mean of all synchronization delay times can be used as the data synchronization deviation of the high-frequency data table. The data synchronization deviation of each high-frequency data table can be obtained in the above manner, wherein the synchronization delay time represents the delay time when using each log information for data synchronization, and the data synchronization deviation represents the synchronization delay when using all log information of the high-frequency data table to synchronize data with the high-frequency data table; finally, the standard deviation of all data synchronization deviations can be used as the data inclination of high-frequency financial information during log playback.

[0078] In some embodiments, determining the data freshness of the high-frequency financial information according to the data skewness and the transaction synchronization record may be implemented by the following steps:

[0079] Obtaining all transaction logs in the transaction synchronization record;

[0080] Determine the log sending time of each transaction log during the data synchronization process through the transaction synchronization record;

[0081] Counting the transaction synchronization records to obtain a synchronization transaction volume;

[0082] Determine the log playback time of the transaction operation in the transaction log according to the synchronization transaction volume and the data inclination;

[0083] The data freshness of the high-frequency financial information is determined according to the sending time of each log and the playback time of the log.

[0084] It should be noted that in this application, data freshness is an indicator to measure the real-time and delay of data in the synchronization process; in specific implementation, first, each log information in the transaction synchronization record can be used as a transaction log to obtain all transaction logs; secondly, for each transaction log, the time difference from the operation time to the synchronization start time in the log information corresponding to the transaction log can be used as the log sending time of the transaction log in the data synchronization process. The log sending time of each transaction log in the data synchronization process can be obtained in the above manner; then, the total number of transaction logs in the transaction synchronization record is counted as the synchronization transaction volume; then, a linear estimation model is initialized, and the synchronization transaction volume and data inclination are used as the linear The estimated parameters in the linear estimation model can be used as the estimation results of the linear estimation model as the log playback time of the transaction operation in the transaction log. If the amount of synchronized transactions is large and the data skewness is small, the log playback time is short. On the contrary, if the amount of synchronized transactions is small and the data skewness is large, the log playback time is long. Finally, for each transaction log, the sum of the log playback time and the log sending time of the transaction log can be used as the data delay when the transaction log is synchronized with the high-frequency data table corresponding to the high-frequency financial information. The data delay when each high-frequency data table is synchronized can be obtained in the above manner, and the mean of all data delays can be used as the data freshness of the high-frequency financial information.

[0085] In step 104, the data reading records of the high-frequency financial information in each data node are obtained, and the conflict between data reading and data synchronization is coupled detected through each data reading record and the log playback site to obtain the conflict coupling entropy between data reading and data synchronization of the corresponding data node.

[0086] It should be noted that in the present application, the data reading record represents the reading data of each data node for high-frequency financial information. In a distributed database system, a data node is the basic unit for storing and managing data. Each data node is usually responsible for the storage, processing and query operations of a part of the data. In the process of data synchronization and high-frequency data management, understanding and managing the performance and synchronization status of data nodes is the key to ensuring system stability and data consistency. In specific implementation, for each data node, the reading data of each high-frequency data table in the high-frequency financial information within a preset time period (the default is 1 week) is obtained in the database. The collection of all read data can be used as the data reading record of the high-frequency financial information in the data node. The data reading record of the high-frequency financial information in each data node can be obtained in the above manner, wherein each read data contains the reading time and the reading data table.

[0087] In some embodiments, coupling detection of the conflict between data reading and data synchronization is performed through each data reading record and the log playback position, and obtaining the conflict coupling entropy between data reading and data synchronization of the corresponding data node can be achieved by the following steps:

[0088] For each data node, obtain the data reading record of the data node;

[0089] Determine the selected read data and the comparison read time corresponding to the log playback location in the data read record;

[0090] Acquire selected reading information corresponding to the data node in the data reading record by using the selected reading data;

[0091] Obtaining the comparison synchronization information corresponding to the data node in the historical log record through the comparison reading time;

[0092] The selected read information is coupled and compared with the comparison synchronization information to obtain the conflict coupling entropy between data reading and data synchronization of the data node, and then obtain the conflict coupling entropy between data reading and data synchronization of each data node.

[0093] It should be noted that the conflict coupling entropy represents the quantitative value of the degree of conflict between the data reading process and the data synchronization process at the data usage moment; in specific implementation, first, for each data node, the data reading record of the data node is obtained; secondly, the high-frequency data table corresponding to the first read data in the data reading record can be used as the selected read data, and the reading moment corresponding to the first read data in the data reading record can be used as the comparison reading moment; then, the set of read data involving the selected read data in the data reading record can be used as the selected read information corresponding to the data node; further, the set of log information whose operation moment in the historical log record is after the comparison reading moment can be used as the comparison synchronization information corresponding to the data node; finally, the number of data items with repeated time in the reading moment in the selected read information and the operation moment in the comparison synchronization information is counted and multiplied by 2 as the conflict value, the sum of the number of data items of the read data in the selected read information and the number of data items of the log information in the comparison synchronization information is obtained as the coupling value, and the ratio of the conflict value to the coupling value can be used as the conflict coupling entropy.

[0094] In step 105, when the high-frequency financial information in the target financial data is synchronously backed up, each data node of the high-frequency financial information is synchronously backed up in parallel based on the data freshness and each conflict coupling entropy.

[0095] In some embodiments, each data node of the high-frequency financial information is backed up in parallel and synchronously based on the data freshness and each conflict coupling entropy, referring to Figure 3The figure is a schematic diagram of the process of parallel synchronous backup in some embodiments of the present application. In this embodiment, the parallel synchronous backup can be implemented by the following steps:

[0096] In step 1051, for each data node, the conflict coupling entropy corresponding to the data node is obtained;

[0097] In step 1052, the synchronization urgency of the high-frequency financial information in the data node is determined by the conflict coupling entropy and the data freshness;

[0098] In step 1053, if the synchronization urgency is greater than a preset urgency threshold, the high-frequency financial information in the data node is synchronously backed up, thereby completing the parallel synchronous backup of each data node.

[0099] In the specific implementation, first, for each data node, the conflict coupling entropy corresponding to the data node is obtained; then, a synchronization model is initialized, and the conflict coupling entropy and data freshness are used as synchronization parameters in the synchronization model. The linear synchronization model is used to perform synchronization emergency verification on the financial data in the data node. The synchronization emergency verification result can be used as the synchronization urgency of the high-frequency financial information in the data node. The synchronization urgency indicates the urgency of data synchronization of the financial information at the current moment. In other embodiments, other models can also be used to determine it. There is no limitation here. It only needs to ensure that the conflict degree between the data reading process and the data synchronization process (that is, the conflict coupling entropy) and the data in the same The influence of delay (i.e., data freshness) in the synchronization process is reduced, thereby reducing the imbalance of thread allocation during data synchronization and improving the data freshness of financial data; finally, if the synchronization urgency is less than or equal to the preset emergency threshold, synchronization is not performed; if the synchronization urgency is greater than the preset emergency threshold, the high-frequency financial information in the data node is synchronously backed up. The parallel synchronous backup of each data node can be completed in the above manner, wherein, in this application, parallel synchronous backup refers to the simultaneous backup of multiple data nodes; in this application, the emergency threshold represents the judgment threshold of the urgency of data synchronization, and the emergency threshold can be preset through historical experience.

[0100] In the present application, when the high-frequency financial information in the target financial data is synchronously backed up, each data node of the high-frequency financial information is synchronously backed up in parallel based on the data freshness and each conflict coupling entropy; wherein, by determining the data freshness of the high-frequency financial information, the real-time performance and delay of the high-frequency financial information in the synchronization process can be obtained. The above steps comprehensively consider the time for the basic log and the transaction log to be sent from the main database to the backup server and the time for data synchronization in the backup server, so as to obtain a more accurate delay of the high-frequency financial information in the synchronization process, which is convenient for the subsequent parallel synchronous backup of the data nodes of the high-frequency financial information in the target financial data, and ensure the data freshness of the financial data in the backup database. Then, the conflict coupling entropy can be determined to obtain the data read The quantitative value of the degree of conflict between the data reading process and the data synchronization process at the data usage time is obtained. The above steps can obtain a more accurate quantitative value of the degree of conflict between the data reading process and the data synchronization process by coupling detection of the conflict (i.e., repeated time points) between the data reading process and the data synchronization process after the log playback site of each data node, so as to facilitate the subsequent comprehensive consideration of the time conflict of data reading and data synchronization when backing up the high-frequency financial information in the target financial data, thereby reducing the delay of data backup and further improving the data freshness of the financial data in the database; in summary, based on the above scheme, the parallel synchronous backup of the data nodes in the financial data can be realized by means of parallel log playback, thereby improving the data freshness of the financial data in the database.

[0101] In addition, in another aspect of the present application, in some embodiments, the present application provides a financial data real-time synchronization backup system, the financial data real-time synchronization backup system also includes a synchronization backup unit, reference Figure 4 , which is a schematic diagram of exemplary hardware and / or software of a synchronization backup unit according to some embodiments of the present application, the synchronization backup unit includes: an acquisition module 201, a processing module 202 and an execution module 203, which are described as follows:

[0102] Acquisition module 201, in this application, acquisition module 201 is mainly used to monitor the log information of target financial data in the database, and then obtain the historical log records of the target financial data;

[0103] Processing module 202, in the present application, is used to filter out high-frequency financial information from the target financial data through the historical log records, and then determine the log playback position of the high-frequency financial information in the data synchronization backup process according to the historical log records;

[0104] It should be noted that the processing module 202 is also used to divide the historical log records into transaction synchronization records and basic synchronization records, determine the data inclination of the high-frequency financial information during log playback through the log playback site and the basic synchronization record, and then determine the data freshness of the high-frequency financial information according to the data inclination and the transaction synchronization record;

[0105] In addition, the processing module 202 is further used to obtain the data reading record of the high-frequency financial information in each data node, and perform coupling detection on the conflict between data reading and data synchronization through each data reading record and the log playback site to obtain the conflict coupling entropy between data reading and data synchronization of the corresponding data node;

[0106] Execution module 203, in the present application, execution module 203 is mainly used for performing parallel synchronization backup of each data node of the high-frequency financial information based on the data freshness and each conflict coupling entropy when the high-frequency financial information in the target financial data is synchronously backed up.

[0107] The above describes in detail the examples of the real-time synchronization and backup method of financial data and related equipment provided by the embodiments of the present application. It can be understood that the corresponding device includes hardware structures and / or software modules corresponding to the execution of each function in order to realize the above functions. It should be easily appreciated by those skilled in the art that, in combination with the units and algorithm steps of each example described in the embodiments disclosed herein, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the present application.

[0108] In some embodiments, the present application also provides a computer device, which includes a memory and a processor, wherein the memory is used to store a computer program, and the processor is used to call and run the computer program from the memory, so that the computer device executes the above-mentioned real-time synchronization and backup method of financial data.

[0109] In some embodiments, reference Figure 5 , the dotted line in the figure indicates that the unit or the module is optional, and the figure is a schematic diagram of the structure of a computer device for implementing a method for real-time synchronization and backup of financial data according to an embodiment of the present application. The method for real-time synchronization and backup of financial data described in the above embodiment can be Figure 5The computer device shown in the figure is implemented, and the computer device includes at least one processor 301, a memory 302 and at least one communication unit 305. The computer device can be a terminal device, a server or a chip.

[0110] The processor 301 may be a general-purpose processor or a special-purpose processor. For example, the processor 301 may be a central processing unit (CPU), which may be used to control the computer device, execute software programs, and process data of the software programs. The computer device may also include a communication unit 305 to implement signal input (reception) and output (transmission).

[0111] For example, the computer device may be a chip, the communication unit 305 may be an input and / or output circuit of the chip, or the communication unit 305 may be a communication interface of the chip, and the chip may be a component of a terminal device, a network device, or other devices.

[0112] For another example, the computer device may be a terminal device or a server, and the communication unit 305 may be a transceiver of the terminal device or the server, or the communication unit 305 may be a transceiver circuit of the terminal device or the server.

[0113] The computer device may include one or more memories 302, on which a program 304 is stored. The program 304 can be executed by the processor 301 to generate instructions 303, so that the processor 301 performs the method described in the above method embodiment according to the instructions 303. Optionally, data (such as a target audit model) can also be stored in the memory 302. Optionally, the processor 301 can also read the data stored in the memory 302, and the data can be stored at the same storage address as the program 304, or the data can be stored at a different storage address from the program 304.

[0114] The processor 301 and the memory 302 may be provided separately or integrated together, for example, integrated on a system on chip (SOC) of the terminal device.

[0115] It should be understood that each step of the above method embodiment can be completed by a hardware-based logic circuit or software-based instructions in the processor 301. The processor 301 can be a CPU, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic devices, such as discrete gates, transistor logic devices, or discrete hardware components.

[0116] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0117] For example, in some embodiments, the present application also provides a computer-readable storage medium, in which instructions or codes are stored. When the instructions or codes are run on a computer, the computer implements the above-mentioned real-time synchronization and backup method of financial data when executing.

[0118] Although the preferred embodiments of the present application have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the present application.

[0119] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A method for real-time synchronous backup of financial data, characterized in that: The steps include: Monitor the log information of the target financial data in the database, and then obtain the historical log records of the target financial data; Filtering out high-frequency financial information from the target financial data through the historical log records, and then determining the log playback position of the high-frequency financial information in the data synchronization backup process according to the historical log records; Dividing the historical log records into transaction synchronization records and basic synchronization records, determining the data inclination of the high-frequency financial information during log playback through the log playback site and the basic synchronization records, and then determining the data freshness of the high-frequency financial information according to the data inclination and the transaction synchronization records; Obtaining data reading records of the high-frequency financial information in each data node, performing coupling detection on the conflict between data reading and data synchronization through each data reading record and the log playback location, and obtaining the conflict coupling entropy between data reading and data synchronization of the corresponding data node; When the high-frequency financial information in the target financial data is synchronously backed up, each data node of the high-frequency financial information is synchronously backed up in parallel based on the data freshness and each conflict coupling entropy; The step of determining the data inclination of the high-frequency financial information during log playback through the log playback location and the basic synchronization record specifically includes: Extracting the operation synchronization queue of each high-frequency data table in the high-frequency financial information at the current moment in the basic synchronization record through the log playback site; Determine the data synchronization deviation of the corresponding high-frequency data table according to each operation synchronization queue; Determine the data skewness of the high-frequency financial information when performing log playback based on all data synchronization deviations; Wherein, determining the data freshness of the high-frequency financial information according to the data inclination and the transaction synchronization record specifically includes: Obtaining all transaction logs in the transaction synchronization record; Determine the log sending time of each transaction log during the data synchronization process through the transaction synchronization record; Counting the transaction synchronization records to obtain a synchronization transaction volume; Determine the log playback time of the transaction operation in the transaction log according to the synchronization transaction volume and the data inclination; Determine the data freshness of the high-frequency financial information according to the sending time of each log and the playback time of the log; The conflict between data reading and data synchronization is coupled detected through each data reading record and the log playback position, and the conflict coupling entropy between data reading and data synchronization of the corresponding data node is obtained, which specifically includes: For each data node, obtain the data reading record of the data node; Determine the selected read data and the comparison read time corresponding to the log playback location in the data read record; Acquire selected reading information corresponding to the data node in the data reading record by using the selected reading data; Obtaining the comparison synchronization information corresponding to the data node in the historical log record through the comparison reading time; The selected read information is coupled and compared with the comparison synchronization information to obtain the conflict coupling entropy between data reading and data synchronization of the data node, and then obtain the conflict coupling entropy between data reading and data synchronization of each data node.

2. The method according to claim 1, characterized in that Determining the log playback location of the high-frequency financial information in the data synchronization backup process according to the historical log records specifically includes: For each high-frequency data table in the high-frequency financial information, obtaining the to-be-synchronized log record of the high-frequency data table in the historical log record; Determine the playback queue of the high-frequency data table at the current moment through the log record to be synchronized, and then obtain the playback queue of each high-frequency data table at the current moment; The log playback position of the high-frequency financial information in the data synchronization backup process is determined according to the log position of the initial playback data corresponding to each playback queue.

3. The method according to claim 1, characterized in that Dividing the historical log records into transaction synchronization records and basic synchronization records specifically includes: Obtaining an operation type identifier for each log message in the historical log record; Obtaining synchronized log records in the historical log records; The synchronized log records are mapped and divided according to each operation type identifier to obtain transaction synchronization records and basic synchronization records.

4. The method according to claim 1, characterized in that The parallel synchronization backup of each data node of the high-frequency financial information based on the data freshness and each conflict coupling entropy specifically includes: For each data node, obtain the conflict coupling entropy corresponding to the data node; Determine the synchronization urgency of high-frequency financial information in the data node through the conflict coupling entropy and the data freshness; If the synchronization urgency is greater than a preset urgency threshold, the high-frequency financial information in the data node is synchronously backed up, thereby completing the parallel synchronous backup of each data node.

5. The method according to claim 1, characterized in that The database is a distributed database.

6. A real-time synchronous backup system for financial data, which uses the method described in any one of claims 1 to 5 to perform real-time synchronous backup of financial data, and the real-time synchronous backup system for financial data includes a synchronous backup unit, characterized in that: The synchronous backup unit comprises: An acquisition module is used to monitor the log information of the target financial data in the database, and then obtain the historical log records of the target financial data; A processing module, configured to filter out high-frequency financial information from the target financial data through the historical log records, and then determine a log playback position of the high-frequency financial information in the data synchronization backup process according to the historical log records; The processing module is also used to divide the historical log records into transaction synchronization records and basic synchronization records, determine the data inclination of the high-frequency financial information during log playback through the log playback site and the basic synchronization record, and then determine the data freshness of the high-frequency financial information according to the data inclination and the transaction synchronization record; The processing module is further used to obtain the data reading record of the high-frequency financial information in each data node, and perform coupling detection on the conflict between data reading and data synchronization through each data reading record and the log playback position, so as to obtain the conflict coupling entropy between data reading and data synchronization of the corresponding data node; The execution module is used to perform parallel synchronization backup on each data node of the high-frequency financial information based on the data freshness and each conflict coupling entropy when the high-frequency financial information in the target financial data is synchronized and backed up.

7. A computer device, characterized in that: The computer device includes a memory and a processor, the memory is used to store a computer program, and the processor is used to call and run the computer program from the memory, so that the computer device executes the real-time synchronization and backup method for financial data according to any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores instructions or codes, and when the instructions or codes are executed on a computer, the computer implements the method for real-time synchronous backup of financial data as described in any one of claims 1 to 5.

Images