Partition mode-based CDC data resource synchronization method and related device thereof

Abstract

The embodiment of the application belongs to the technical field of financial science and technology, is applied to a financial industry data resource synchronization scene, and relates to a CDC data resource synchronization method based on a partition mode and related equipment thereof. When data source end data resources are synchronized to a target end, different partition modes are selected for partition processing of a target data table through a screening condition, so that the partition processing is more scientific and conforms to synchronization requirements, previous consistent partition by using row numbers and column numbers is avoided, and the end-to-end data resource synchronization scene is more consistent. Table field information corresponding to specific data content during partition is introduced into a table name of a subform, so that when data resource synchronization is performed, different names are screened through the table field information, a subform corresponding to the different names containing the table field information is screened out, data resource synchronization is performed, searching and positioning are more rapid, searching of a large wide table is avoided, and consumption of network processing resources is reduced.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of financial technology, and is applied to a financial industry data resource synchronization scene, and in particular relates to a CDC data resource synchronization method based on a partition mode and a related device thereof. BACKGROUND

[0002] At present, the industry often uses a CDC (Change Data Capture, change data capture) data scheme to complete synchronization of data between ends in the field of big data. The scheme obtains and parses operation record logs of data in a database, including operations such as insertion, update, and deletion of data, records the data operation record logs in sequence, and transmits the data operation record logs to a downstream platform of the database through a stream system, and the downstream completes replay of the data operation actions to realize real-time or near-real-time synchronization of data between ends, so as to efficiently provide the source end data for processing and application of the downstream.

[0003] The CDC scheme has obvious advantages in data timeliness and data accuracy. However, its disadvantages are also very obvious, especially in the transmission of financial business data, that is, the complete log needs to be recorded and transmitted for each operation of the financial business data between ends. Since the financial business data is often large and complex, this causes great performance pressure on the source end database system and the stream system used for log message transmission. In actual production application scenarios, most financial business database administrators will reduce the amount of CDC data information, and only transmit necessary key financial business data to reduce the performance pressure caused in the process of analyzing and transmitting the financial business data operation log. However, the lack of complete financial business data operation log information causes great trouble for the system downstream of the database to complete data synchronization. The downstream system can only locate and update the data through the primary key, and needs to consume a large amount of computing resources to complete the scanning and positioning of the data. The above scheme only transfers the performance pressure to the downstream of the database. Therefore, the prior art still lacks a synchronization scheme that can balance the performance consumption of all systems in the whole link in the synchronization of CDC data resources under a large data scale. SUMMARY

[0004] The purpose of the embodiments of the present application is to provide a CDC data resource synchronization method based on a partition mode and a related device thereof to solve the problem that the prior art still lacks a synchronization scheme that can balance the performance consumption of all systems in the whole link in the synchronization of CDC data resources under a large data scale.

[0005] To solve the above technical problems, the embodiments of the present application provide a CDC data resource synchronization method based on a partition mode, which adopts the following technical scheme:

[0006] A CDC data resource synchronization method based on a partition mode, comprising the following steps:

[0007] Step 201, obtaining data operation record logs of a data source end, performing analysis, and obtaining an analysis result;

[0008] Step 202, identifying table field information involved in data resource synchronization at a target end according to the analysis result;

[0009] Step 203, taking any table field information in the table field information involved as search information, and finding out all data tables to be synchronized in the target end;

[0010] Step 204, pre-processing all data tables to be synchronized according to a preset pre-processing mode, and obtaining a pre-processing result;

[0011] Step 205, distinguishingly naming the pre-processing result according to a preset naming rule;

[0012] Step 206, synchronizing target data resources to the target end through the distinguishing naming.

[0013] Further, the step of obtaining data operation record logs of a data source end, performing analysis, and obtaining an analysis result specifically comprises:

[0014] Step 301, obtaining record content in the data operation record logs line by line;

[0015] Step 302, identifying whether there is table field information in the record content in the data operation record logs of the current line according to a preset table field information set;

[0016] Step 303, if there is table field information in the record content in the data operation record logs of the current line, screening out the table field information and data resource information corresponding to the table field information, and caching the table field information and the data resource information corresponding to the table field information in a key-value pair data format, wherein the data resource information corresponding to the table field information includes specific data content changed, and row and column information of the corresponding data table after the specific data content is changed;

[0017] Step 304, if there is no table field information in the record content in the data operation record logs of the current line, obtaining record content of the next line in the data operation record logs, and repeating steps 302 to 304;

[0018] Step 305, until the current line is the last line of the data operation record logs, the analysis is completed.

[0019] Further, the step of caching the table field information and the data resource corresponding to the table field information in a key-value pair data format according to the key value, specifically comprises:

[0020] Taking the table field information as a search field, it is identified whether the key-value pair data corresponding to the table field information already exists in the preset cache area, wherein the format of the key-value pair data is specifically

[0021] If the key-value pair data corresponding to the table field information already exists in the preset cache area, the data resource information is obtained to update the key-value pair data;

[0022] If the key-value pair data corresponding to the table field information does not exist in the preset cache area, the table field information and the data resource information corresponding to the table field information are written into the cache area in a key-value pair format.

[0023] Further, the step of preprocessing all the data tables to be synchronized according to a preset preprocessing mode, specifically comprises:

[0024] Step 401, respectively acquiring the row and column values of each data table to be synchronized;

[0025] Step 402, according to a preset filtering rule and the row and column values of each data table to be synchronized, filtering out the data tables that need to be partitioned;

[0026] Step 403, according to a preset partitioning rule, partitioning all the data tables that need to be partitioned to obtain a sub-table after partitioning;

[0027] Step 404, respectively acquiring the row and column values of each sub-table, and iteratively executing steps 402 to 404 until no table needs to be partitioned, and terminating the iteration;

[0028] Step 405, obtaining the final sub-table obtained by partitioning as a preprocessing result.

[0029] Further, the step of filtering out the data tables that need to be partitioned according to a preset filtering rule and the row and column values of each data table to be synchronized, specifically comprises:

[0030] Acquiring a maximum row and column threshold of the data tables that do not need to be partitioned;

[0031] Comparing the row and column values of each data table to be synchronized with the maximum row and column threshold;

[0032] If the row and column values of the current target data table in the data tables to be synchronized are greater than the preset maximum row and column threshold, the target data table is partitioned.

[0033] If the row and column values of the current target data table in the various data tables to be synchronized are not greater than the preset maximum row and column value threshold, the target data table is not subjected to partition processing.

[0034] Further, if the row and column values of the current target data table in the various data tables to be synchronized are greater than the preset maximum row and column value threshold, the target data table is subjected to partition processing, specifically including:

[0035] determining whether there are several specific data content with the same table field information in the current target data table;

[0036] If there are, the first partition method is used to partition the current target data table, wherein the first partition method specifically extracts the rows corresponding to the specific data content and arranges all the extracted rows into one partition;

[0037] If there are not, the second partition method is used to partition the current target data table, wherein the second partition method specifically partitions the current target data table according to the maximum row value threshold or / and the maximum column value threshold.

[0038] Further, the step of distinguishing the naming of the preprocessing result according to the preset naming rule specifically includes:

[0039] If the partition method corresponding to the preprocessing result is the first partition method, the table name of the corresponding data table before this partition processing and the specific data content at this partition processing are obtained, and the table field information corresponding to the table name and the specific data content is spliced as the distinguished naming of the preprocessing result;

[0040] If the partition method corresponding to the preprocessing result is the second partition method, the table name of the corresponding data table before this partition processing and the partition number at this partition processing are obtained, and the table name and the partition number are spliced as the distinguished naming of the preprocessing result;

[0041] The step of synchronizing the target data resource to the target end through the distinguished naming specifically includes:

[0042] determining whether the table field information involved in the data resource synchronization of the target end is contained in the string corresponding to the distinguished naming;

[0043] If the string contains the table field information, the subform corresponding to the distinguished naming is taken as a search field, and the cells in the subform for data resource synchronization are searched and synchronized;

[0044] If the table field information is not included in the string, it is determined whether the table field information is table field information of a subform corresponding to the distinguished naming;

[0045] If the table field information is table field information of a subform corresponding to the distinguished naming, the subform corresponding to the distinguished naming is taken as a search domain, and a cell in the subform that performs data resource synchronization is searched out for a synchronization operation.

[0046] If the table field information is not table field information of a subform corresponding to the distinguished naming, the subform is not operated.

[0047] To solve the above technical problems, the embodiment of the application further provides a CDC data resource synchronization device based on a partition mode, which adopts the technical scheme as follows:

[0048] A CDC data resource synchronization device based on a partition mode, comprising:

[0049] A log analysis module, configured to acquire data operation record logs of a data source end, perform analysis, and acquire an analysis result;

[0050] A table field information identification module, configured to identify table field information involved in data resource synchronization at a target end according to the analysis result;

[0051] A table to be synchronized searching module, configured to take any table field information in the table field information involved as search information, and search out all data tables to be synchronized in the target end;

[0052] A form preprocessing module, configured to perform preprocessing on all data tables to be synchronized according to a preset preprocessing mode, and acquire a preprocessing result;

[0053] A distinguished naming module, configured to perform distinguished naming on the preprocessing result according to a preset naming rule;

[0054] A data resource synchronization module, configured to synchronize target data resources to the target end through the distinguished naming.

[0055] To solve the above technical problems, the embodiment of the application further provides a computer device, which adopts the technical scheme as follows:

[0056] A computer device, comprising a memory and a processor, wherein the memory stores computer readable instructions, and the processor implements the steps of the CDC data resource synchronization method based on a partition mode when executing the computer readable instructions.

[0057] To solve the above technical problems, the embodiment of the present application also provides a computer readable storage medium, which adopts the technical scheme as follows:

[0058] A computer readable storage medium, which stores computer readable instructions, and the computer readable instructions are executed by a processor to implement the steps of the CDC data resource synchronization method based on the partition mode.

[0059] Compared with the prior art, the embodiment of the present application has the following beneficial effects:

[0060] The CDC data resource synchronization method based on the partition mode has the following beneficial effects: when the data resource synchronization from the data source end to the target end is performed, the data tables in the target end are preprocessed by partitioning, so that a large amount of data resources is avoided to be searched when the data resource synchronization is performed, and the consumption of network processing resources is reduced; by using certain screening conditions, different partitioning modes are selected to perform partitioning on the target data tables, so that the partitioning is more scientific and meets the data resource synchronization requirements; the previous partitioning by using row numbers and column numbers is avoided, and the end-to-end data resource synchronization scenario is more suitable; by introducing the table field information corresponding to the specific data content in the first partitioning mode into the table name of the subform, when the data resource synchronization is performed, the subform corresponding to the distinguished naming containing the table field information is screened by screening the distinguished naming by using the table field information, and then the data resource synchronization is performed, so that the searching and positioning are more rapid, and the searching of the wide table is avoided. BRIEF DESCRIPTION OF DRAWINGS

[0061] In order to more clearly illustrate the schemes in the present application, the drawings needed in the description of the embodiments of the present application will be briefly introduced as follows. Obviously, the drawings in the following description are some embodiments of the present application, and other drawings can be obtained by those skilled in the art without any creative effort on the basis of these drawings.

[0062] Figure 1 is an exemplary system architecture diagram to which the present application can be applied;

[0063] Figure 2A flow chart of one embodiment of the partition pattern based CDC data resource synchronization method according to the present application;

[0064] Figure 3 is Figure 2 A flow chart of one embodiment of the step 201 shown in FIG. 2;

[0065] Figure 4 is Figure 2 A flow chart of one embodiment of the step 204 shown in FIG. 2;

[0066] Figure 5 is Figure 4 A flow chart of one embodiment of the step 402 shown in FIG. 4;

[0067] Figure 6 is Figure 4 A flow chart of one embodiment of the step 403 shown in FIG. 4;

[0068] Figure 7 is Figure 2 A flow chart of one embodiment of the step 205 shown in FIG. 2;

[0069] Figure 8 is Figure 2 A flow chart of one embodiment of the step 206 shown in FIG. 2;

[0070] Figure 9 A structural schematic diagram of one embodiment of the partition pattern based CDC data resource synchronization apparatus according to the present application;

[0071] Figure 10 A structural schematic diagram of one embodiment of the computer device according to the present application. DETAILED DESCRIPTION

[0072] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs; the terminology used in the description herein is for describing particular embodiments only and is not intended to be limiting of the application; the description and the drawings are to be regarded as illustrative in nature and are not intended to limit the application; the terminology used in the description and the claims of the application and the above description of the drawings includes the terms specifically mentioned above, as well as their derivatives.

[0073] Reference to an "embodiment" herein means that a particular feature, structure, or characteristic described in connection with the embodiment can be included in at least one embodiment of the application. The appearances of the phrase in various places in the specification are not necessarily all referring to the same embodiment, nor are they necessarily mutually exclusive of one another. As will be apparent to those of ordinary skill in the art, embodiments described herein can be combined with other embodiments.

[0074] For better understanding of the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings.

[0075] As shown in Figure 1 The system architecture 100 can include terminal devices 101, 102, 103, a network 104, and a server 105. The network 104 is a medium for providing a communication link between the terminal devices 101, 102, 103 and the server 105. The network 104 can include various connection types, such as wired, wireless communication links, or optical fiber cables, and the like.

[0076] A user can use the terminal devices 101, 102, 103 to interact with the server 105 through the network 104 to receive or send messages, and the like. Various communication client applications can be installed on the terminal devices 101, 102, 103, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, and the like.

[0077] The terminal devices 101, 102, 103 can be various electronic devices with display screens and supporting web browsing, including but not limited to smart phones, tablet computers, e-book readers, MP3 (Moving Picture Experts Group Audio Layer III) players, MP4 (Moving Picture Experts Group Audio Layer IV) players, laptop computers, desktop computers, and the like.

[0078] The server 105 can be a server providing various services, such as a background server providing support for a page displayed on the terminal devices 101, 102, 103.

[0079] It should be noted that the CDC data resource synchronization method based on the partition mode provided in the embodiments of the present application is generally executed by a server / terminal device, and accordingly, the CDC data resource synchronization apparatus based on the partition mode is generally arranged in a server / terminal device.

[0080] It should be understood that,Figure 1 The number of terminal devices, networks and servers in the system is only illustrative. Any number of terminal devices, networks and servers can be provided according to implementation needs.

[0081] With reference to Figure 2 , a flow chart of one embodiment of the partition mode based CDC data resource synchronization method according to the present application is shown. The partition mode based CDC data resource synchronization method includes the following steps:

[0082] Step 201, obtaining data operation record logs of a data source end, parsing and obtaining a parsing result.

[0083] In this embodiment, the data source end is a financial industry data source end, for example, an insurance business data source end, a claim settlement business data source end, the data source end includes a data warehouse storing pre-set business data, the pre-set business data refers to business data before the present processing.

[0084] With reference to Figure 3 , Figure 3 is Figure 2 a flow chart of one specific embodiment of step 201, including:

[0085] Step 301, obtaining record content in the data operation record logs line by line;

[0086] Step 302, identifying whether there is table field information in the record content in the data operation record logs of the current line according to a preset table field information set;

[0087] Step 303, if there is table field information in the record content in the data operation record logs of the current line, screening out the table field information and data resource information corresponding to the table field information, and caching the table field information and the data resource information corresponding to the table field information in a key-value pair data format;

[0088] In this embodiment, the data resource information corresponding to the table field information includes specific data content that has changed, and row and column information in the corresponding data table after the specific data content has changed.

[0089] In the embodiment, the step of caching the table field information and the data resource corresponding to the table field information in the key-value pair data format comprises: taking the table field information as a search field, identifying whether the key-value pair data corresponding to the table field information exists in a preset cache area; if the key-value pair data corresponding to the table field information exists in the preset cache area, updating the key-value pair data with the data resource information; and if the key-value pair data corresponding to the table field information does not exist in the preset cache area, writing the table field information and the data resource information corresponding to the table field information into the cache area in the key-value pair format.

[0090] In the embodiment, the format of the key-value pair data is

[0091] If the table field information does not exist in the record content in the data operation record log of the current row, the next row record content in the data operation record log is obtained, and the steps 302 to 304 are repeatedly executed.

[0092] The parsing is completed until the current row is the last row in the data operation record log.

[0093] According to the analysis result, the table field information involved in the data resource synchronization at the target end is identified.

[0094] Taking any table field information in the table field information involved as search information, all data tables to be synchronized in the target end are found out.

[0095] According to a preset preprocessing mode, all data tables to be synchronized are preprocessed, and a preprocessing result is obtained.

[0096] In the embodiment, when the data resource synchronization from the data source end to the target end is performed, the data tables in the target end are preprocessed by partitioning, so that the retrieval of a large amount of data resources during the data resource synchronization is avoided, and the consumption of network processing resources is reduced.

[0097] With reference to Figure 4 , Figure 4 is Figure 2 a flow chart of one specific embodiment of the step 204, comprising:

[0098] The row and column values of each data table to be synchronized are obtained respectively.

[0099] According to a preset filtering rule and the row and column values of each data table to be synchronized, the data tables needing partitioning processing are filtered out.

[0100] With reference to both Figure 5 , Figure 5 is Figure 4 a flow chart of one specific embodiment of step 402, comprising:

[0101] Step 501, obtaining a preset maximum row-column threshold of a data table that does not need partition processing;

[0102] Step 502, comparing the row-column value of each data table to be synchronized with the maximum row-column threshold;

[0103] Step 503, if the row-column value of the current target data table in the each data table to be synchronized is greater than the preset maximum row-column threshold, performing partition processing on the target data table;

[0104] Step 504, if the row-column value of the current target data table in the each data table to be synchronized is not greater than the preset maximum row-column threshold, not performing partition processing on the target data table.

[0105] Step 403, performing partition processing on all data tables that need to be partitioned according to a preset partition rule, to obtain a sub-table after partition processing;

[0106] With reference to both Figure 6 , Figure 6 is Figure 4 a flow chart of one specific embodiment of step 403, comprising:

[0107] Step 601, judging whether there are several specific data contents with the same table field information in the current target data table;

[0108] Step 602, if there are, performing partition processing on the current target data table by using a first partition method, wherein the first partition method specifically extracts the rows corresponding to the specific data contents and arranges all the extracted rows into one partition;

[0109] Step 603, if there are not, performing partition processing on the current target data table by using a second partition method, wherein the second partition method specifically performs partition processing on the current target data table according to the maximum row value threshold or / and the maximum column value threshold.

[0110] By using certain screening conditions, different partition methods are selected to perform partition processing on the target data table, so that the partition processing is more scientific and meets the data resource synchronization demand, and avoids the previous consistent partition by using row numbers and column numbers, which is more in line with the end-to-end data resource synchronization scenario.

[0111] Step 404, respectively, obtain the row and column values of each sub-form, and iteratively execute steps 402 to 404 until no form needs to be processed by partitioning, and terminate iteration;

[0112] Step 405, obtain the final sub-form obtained by the partitioning processing as the preprocessing result.

[0113] Step 205, according to the preset naming rule, the preprocessing result is named differently.

[0114] With reference to Figure 7 , Figure 7 is Figure 2 a flow chart of one specific embodiment of step 205, including:

[0115] Step 701, if the partitioning manner corresponding to the preprocessing result is the first partitioning manner, obtain the table name of the corresponding data table before the current partitioning processing and the specific data content at the time of the current partitioning processing, and splice the table field information corresponding to the table name and the specific data content as the differential naming of the preprocessing result;

[0116] Step 702, if the partitioning manner corresponding to the preprocessing result is the second partitioning manner, obtain the table name of the corresponding data table before the current partitioning processing and the partitioning number at the time of the current partitioning processing, and splice the table name and the partitioning number as the differential naming of the preprocessing result.

[0117] By introducing the table field information corresponding to the specific data content in the first partitioning manner into the table name of the sub-form, when performing data resource synchronization, the differential naming is screened through the table field information, the sub-form corresponding to the differential naming containing the table field information is screened out, data resource synchronization is performed, and it is more quick to find and locate, and searching for a large wide table is avoided.

[0118] Step 206, synchronize the target data resource to the target end through the differential naming.

[0119] With reference to Figure 8 , Figure 8 is Figure 2 a flow chart of one specific embodiment of step 206, including:

[0120] Step 801, judge whether the table field information involved in the data resource synchronization of the target end is contained in the string corresponding to the differential naming;

[0121] Step 802, if the table field information is contained in the string, the sub-form corresponding to the differential naming is taken as a search domain, and the cells in the sub-form for data resource synchronization are searched out for synchronization operation;

[0122] Step 803, if the table field information is not contained in the string, judging whether the table field information is the table field information of the sub-form corresponding to the distinguished naming;

[0123] Step 804, if the table field information is the table field information of the sub-form corresponding to the distinguished naming, taking the sub-form corresponding to the distinguished naming as a search domain to search for the cells in the sub-form for data resource synchronization to perform synchronization operation;

[0124] Step 805, if the table field information is not the table field information of the sub-form corresponding to the distinguished naming, not performing operation on the sub-form.

[0125] In addition, after the sub-form is determined through the table field information, the CDC data resource synchronization method based on the partition mode further comprises: acquiring the key-value pair data in the preset cache area according to the table field information; acquiring the row value or / and column value of the changed specific data content by analyzing the key-value pair data; and locating the corresponding cell position of the changed specific data content in the sub-form according to the row value or / and column value.

[0126] Specifically, if the sub-form is a form generated in the first partition mode, only the row value of the changed specific data content is needed to locate the corresponding cell position of the changed specific data content in the sub-form. If the sub-form is a form generated in the second partition mode, the row value and the column value of the changed specific data content are needed to locate the corresponding cell position of the changed specific data content in the sub-form.

[0127] The application obtains data operation record logs of a data source end, performs analysis, and obtains an analysis result; according to the analysis result, table field information involved in data resource synchronization at a target end is identified; any table field information in the table field information involved is taken as search information to find all data tables to be synchronized in the target end; all the data tables to be synchronized are preprocessed according to a preset preprocessing mode to obtain a preprocessing result; the preprocessing result is distinguishedly named according to a preset naming rule; and the target data resource is synchronized to the target end through the distinguished naming. When data resource synchronization is performed from the data source end to the target end, the data tables in the target end are preprocessed by partitioning, which avoids searching for a large amount of data resources during data resource synchronization, reduces consumption of network processing resources, and selects different partitioning modes for partitioning the target data tables through certain screening conditions, so that the partitioning is more scientific and meets the data resource synchronization requirements, avoids the conventional partitioning by using row numbers and column numbers, is more suitable for end-to-end data resource synchronization scenarios, and through introducing the table field information corresponding to the specific data content in the first partitioning mode into the table name of the subform, the subform corresponding to the distinguished name containing the table field information is screened through the table field information for the distinguished naming during data resource synchronization, data resource synchronization is performed, the data tables are more quickly searched and located, and searching for a large wide table is avoided.

[0128] The embodiments of the application can acquire and process related data based on artificial intelligence technology. Artificial intelligence (AI) is the use of digital computers or computer-controlled machines to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use the knowledge to obtain the best results.

[0129] Artificial intelligence basic technologies generally include technologies such as sensors, special artificial intelligence chips, cloud computing, distributed storage, big data processing, operation / interaction systems, and mechatronics. Artificial intelligence software technologies mainly include computer vision technology, robot technology, biometric technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0130] In the embodiment of the present application, when data resource synchronization is performed from the data source end to the target end, different partitioning methods are selected according to certain filtering conditions to partition the target data table, so that the partitioning is more scientific and meets the data resource synchronization requirements, avoiding the use of row numbers and column numbers for partitioning in the past, which is more in line with the end-to-end data resource synchronization scenario. By introducing the table field information corresponding to the specific data content in the first partitioning method into the table name of the subtable, when data resource synchronization is performed, the subtable corresponding to the different naming containing the table field information is filtered through the table field information, and the data resource synchronization is performed, which is more quickly searched and positioned, and the search for a large wide table is avoided.

[0131] Further referring to Figure 9 , as an implementation of the method shown in the above Figure 2 , the present application provides an embodiment of a CDC data resource synchronization device based on a partitioning mode. The device embodiment corresponds to the method embodiment shown in Figure 2 , and the device can be applied to various electronic devices.

[0132] As shown in Figure 9 , the CDC data resource synchronization device based on a partitioning mode 900 comprises a log analysis module 901, a table field information identification module 902, a table to be synchronized searching module 903, a table preprocessing module 904, a different naming module 905, and a data resource synchronization module 906.

[0133] Among them:

[0134] The log analysis module 901 is configured to obtain data operation record logs of the data source end, analyze the logs, and obtain an analysis result.

[0135] The table field information identification module 902 is configured to identify table field information involved in data resource synchronization at the target end according to the analysis result.

[0136] The table to be synchronized searching module 903 is configured to search for all data tables to be synchronized in the target end by taking any table field information in the table field information as search information.

[0137] The table preprocessing module 904 is configured to preprocess all data tables to be synchronized according to a preset preprocessing method, and obtain a preprocessing result.

[0138] The different naming module 905 is configured to differently name the preprocessing result according to a preset naming rule.

[0139] The data resource synchronization module 906 is configured to synchronize the target data resource to the target end through the different naming.

[0140] The application obtains data operation record logs of a data source end, performs analysis, and obtains an analysis result; according to the analysis result, table field information involved in data resource synchronization at a target end is identified; any table field information in the table field information involved is taken as search information to find all data tables to be synchronized in the target end; according to a preset preprocessing mode, all the data tables to be synchronized are preprocessed to obtain a preprocessing result; the preprocessing result is distinguished according to a preset naming rule; and the target data resource is synchronized to the target end through the distinguished naming. When data resource synchronization is performed from the data source end to the target end, by performing partition processing on the data tables in the target end in advance, searching for a large amount of data resources during data resource synchronization is avoided, consumption of network processing resources is reduced, by a certain screening condition, different partition modes are selected to perform partition processing on the target data tables, so that the partition processing is more scientific and meets the data resource synchronization demand, and the conventional partition by using row numbers and column numbers is avoided, which is more in line with the end-to-end data resource synchronization scene, by introducing the table field information corresponding to the specific data content in the first partition mode into the table name of the subform, when data resource synchronization is performed, the distinguished naming is screened through the table field information, the subform corresponding to the distinguished naming containing the table field information is screened out, data resource synchronization is performed, and more rapid searching and positioning are achieved, and searching for a large wide table is avoided.

[0141] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by computer readable instructions instructing related hardware, and the computer readable instructions can be stored in a computer readable storage medium. When the program is executed, the processes of the above-mentioned embodiments of each method can be included. The storage medium can be a non-volatile storage medium such as a magnetic disc, an optical disc, a read-only memory (ROM), or a random access memory (RAM).

[0142] It should be understood that, although each step in the flowchart of the accompanying drawings is displayed in sequence according to the direction of the arrow, these steps are not necessarily executed in sequence according to the direction of the arrow. Unless otherwise specified herein, the execution of these steps is not strictly limited in sequence, and they can be executed in other sequences. Moreover, at least part of the steps in the flowchart of the accompanying drawings can include multiple sub-steps or multiple stages, which are not necessarily executed at the same time, but can be executed at different times, and the execution sequence is not necessarily sequential, but can be executed in rotation or alternation with at least part of other steps or sub-steps or stages of other steps.

[0143] To solve the above technical problems, the embodiments of the present application further provide a computer device. For details, please refer to Figure 10 , Figure 10 The basic structure block diagram of the computer device of the present embodiment is shown in the figure.

[0144] The computer device 10 comprises a memory 10a, a processor 10b, and a network interface 10c, which are connected to each other through a system bus. It should be noted that only the computer device 10 with components 10a-10c is shown in the figure, but it should be understood that it is not required to implement all the components shown, and more or fewer components can be alternatively implemented. Among them, those skilled in the art can understand that the computer device herein is a device capable of automatically performing numerical calculation and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to microprocessors, application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), digital signal processors (DSPs), embedded devices, etc.

[0145] The computer device can be a desktop computer, a notebook computer, a palm computer, a cloud server, and the like. The computer device can interact with the user through a keyboard, a mouse, a remote controller, a touchpad, a voice control device, and the like.

[0146] The memory 10a includes at least one type of readable storage medium, such as a flash memory, a hard disk, a multimedia card, a card-type memory (e.g., SD or DX memory, etc.), a random access memory (RAM), a static random access memory (SRAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a programmable read-only memory (PROM), a magnetic memory, a magnetic disk, an optical disk, etc. In some embodiments, the memory 10a can be an internal storage unit of the computer device 10, such as a hard disk or a memory of the computer device 10. In other embodiments, the memory 10a can also be an external storage device of the computer device 10, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the computer device 10. Of course, the memory 10a can also include both the internal storage unit and the external storage device of the computer device 10. In this embodiment, the memory 10a is generally used to store an operating system and various application software installed on the computer device 10, such as computer readable instructions of the CDC data resource synchronization method based on the partition mode, etc. In addition, the memory 10a can also be used to temporarily store various data that have been output or will be output.

[0147] The processor 10b can be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chip in some embodiments. The processor 10b is generally used to control the overall operation of the computer device 10. In this embodiment, the processor 10b is used to run computer readable instructions or process data stored in the memory 10a, such as computer readable instructions of the CDC data resource synchronization method based on the partition mode.

[0148] The network interface 10c can include a wireless network interface or a wired network interface, and is generally used to establish a communication connection between the computer device 10 and other electronic devices.

[0149] The computer device provided in the embodiment belongs to the technical field of financial technology and is applied to the scene of financial industry data resource synchronization. The application obtains data operation record logs of a data source end, analyzes the data operation record logs, and obtains an analysis result. According to the analysis result, table field information involved in data resource synchronization at a target end is identified. Any table field information in the table field information involved is taken as search information to find all data tables to be synchronized in the target end. According to a preset preprocessing mode, all the data tables to be synchronized are preprocessed to obtain a preprocessing result. The preprocessing result is distinguished according to a preset naming rule. The target data resource is synchronized to the target end through the distinguished naming. When data resource synchronization is performed from the data source end to the target end, the data tables in the target end are preprocessed by partitioning, which avoids searching for a large amount of data resources during data resource synchronization, reduces the consumption of network processing resources, and selects different partitioning modes for partitioning the target data tables through certain screening conditions, so that the partitioning is more scientific and meets the data resource synchronization requirements. The previous consistent partitioning by using row numbers and column numbers is avoided, which is more in line with the end-to-end data resource synchronization scene. The table field information corresponding to the specific data content in the first partitioning mode is introduced into the table name of the subform, so that the distinguished naming is screened through the table field information during data resource synchronization, the subform corresponding to the distinguished naming containing the table field information is screened out, and data resource synchronization is performed, which is more rapid in searching and positioning and avoids searching for a large wide table.

[0150] The application also provides another implementation, namely providing a computer readable storage medium, the computer readable storage medium stores computer readable instructions, the computer readable instructions can be executed by a processor to make the processor execute the steps of the partition mode based CDC data resource synchronization method as described above.

[0151] The computer readable storage medium provided in the embodiment belongs to the technical field of financial technology and is applied to a financial industry data resource synchronization scene. The application obtains data operation record logs of a data source end, performs analysis, and obtains an analysis result. According to the analysis result, table field information involved in data resource synchronization at a target end is identified. Any table field information in the table field information is taken as search information to find all data tables to be synchronized in the target end. According to a preset preprocessing mode, all the data tables to be synchronized are preprocessed to obtain a preprocessing result. The preprocessing result is distinguished according to a preset naming rule. The target data resource is synchronized to the target end through the distinguished naming. When data resource synchronization is performed from the data source end to the target end, the data tables in the target end are preprocessed by partitioning, which avoids searching for a large amount of data resources during data resource synchronization, reduces the consumption of network processing resources, and selects different partitioning modes for partitioning the target data tables through certain screening conditions, so that the partitioning is more scientific and meets the data resource synchronization requirements. The previous consistent partitioning by using row numbers and column numbers is avoided, which is more in line with the end-to-end data resource synchronization scene. The table field information corresponding to the specific data content in the first partitioning mode is introduced into the table name of the subform, so that the distinguished naming is screened through the table field information during data resource synchronization, the subform corresponding to the distinguished naming containing the table field information is screened out, and data resource synchronization is performed, which is more rapid in searching and positioning and avoids searching for a large wide table.

[0152] Through the description of the above implementation manner, those skilled in the art can clearly understand that the above-mentioned embodiment method can be realized by means of software and a necessary general hardware platform, of course, it can also be realized by hardware, but in many cases, the former is a better implementation manner. Based on such understanding, the technical solutions of the application can be embodied in the form of a software product, which is stored in a storage medium (such as a ROM / RAM, a magnetic disk, an optical disk), and includes a plurality of instructions for making a terminal device (which can be a mobile phone, a computer, a server, an air conditioner, or a network device) execute the method described in each embodiment of the application.

[0153] Obviously, the above-described embodiments are only some embodiments but not all the embodiments of the present application, the preferred embodiments of the present application are shown in the drawings, but do not limit the patent scope of the present application. The present application can be implemented in many different forms, and conversely, the purpose of providing these embodiments is to make the disclosure of the present application more thorough and comprehensive. Although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing specific embodiments, or make equivalent replacements to some technical features therein. Any equivalent structure made by using the content of the specification and drawings, directly or indirectly applied to other related technical fields, is also within the patent protection scope of the present application.

Claims

1. A CDC data resource synchronization method based on a partition mode, characterized in that, The method comprises the following steps: Step 201, obtaining data operation record logs of a data source end, performing analysis, and obtaining an analysis result; Step 202, identifying table field information involved in data resource synchronization at a target end according to the analysis result; Step 203, taking any table field information in the table field information as search information, and finding all data tables to be synchronized in the target end; Step 204, pre-processing all data tables to be synchronized according to a preset pre-processing mode, and obtaining a pre-processing result, specifically comprising: Step 401, obtaining row and column values of each data table to be synchronized respectively; Step 402, screening data tables requiring partition processing according to a preset screening rule and the row and column values of each data table to be synchronized; Step 403, performing partition processing on all data tables requiring partition processing according to a preset partition rule, and obtaining sub-tables after partition processing; Step 404, obtaining row and column values of each sub-table, and iteratively executing steps 402 to 404 until no table requires partition processing, and terminating iteration; Step 405, obtaining final sub-tables obtained through partition processing as the pre-processing result; Step 205, distinguishingly naming the pre-processing result according to a preset naming rule; Step 206, synchronizing target data resources to the target end through the distinguishing naming.

2. The CDC data resource synchronization method based on partition mode according to claim 1, characterized in that, The step of obtaining data operation record logs of a data source end, performing analysis, and obtaining an analysis result specifically comprises: Step 301, obtaining record content in the data operation record logs line by line; Step 302, identifying whether there is table field information in the record content in the data operation record logs of the current line according to a preset table field information set; Step 303, if there is table field information in the record content in the data operation record logs of the current line, screening the table field information and data resource information corresponding to the table field information, and caching the table field information and the data resource information corresponding to the table field information in a key-value pair data format, wherein the data resource information corresponding to the table field information comprises specific data content that has changed, and row and column information of the specific data content in a corresponding data table after the specific data content has changed; Step 304, if there is no table field information in the record content in the data operation record logs of the current line, obtaining next line record content in the data operation record logs, and repeating steps 302 to 304; Step 305, until the current line is the last line of the data operation record logs, the analysis is completed.

3. The CDC data resource synchronization method based on partition mode according to claim 2, characterized in that, The step of caching the table field information and the data resource corresponding to the table field information in a key-value pair data format specifically comprises: With the table field information as a search field, it is identified whether the key-value pair data corresponding to the table field information exists in a preset cache area, wherein the format of the key-value pair data is specifically ; If the key-value pair data corresponding to the table field information already exists in the preset cache area, updating the key-value pair data with the data resource information. If the key-value pair data corresponding to the table field information does not exist in the preset cache area, the table field information and the data resource information corresponding to the table field information are written into the cache area in the key-value pair format.

4. The CDC data resource synchronization method based on partition mode according to claim 1, characterized in that, The step of screening the data table needing to be processed by partitioning according to the preset screening rule and the row and column values of each data table to be synchronized specifically includes: Obtaining the maximum row and column threshold of the data table not needing to be processed by partitioning; Comparing the row and column values of each data table to be synchronized with the maximum row and column threshold; If the row and column values of the current target data table in the each data table to be synchronized are greater than the preset maximum row and column threshold, the target data table is processed by partitioning; If the row and column values of the current target data table in the each data table to be synchronized are not greater than the preset maximum row and column threshold, the target data table is not processed by partitioning.

5. The CDC data resource synchronization method based on partition mode according to claim 4, characterized in that, The step of processing the target data table by partitioning if the row and column values of the current target data table in the each data table to be synchronized are greater than the preset maximum row and column threshold specifically includes: Judging whether there are several specific data contents of the same table field information in the current target data table; If yes, the first partitioning mode is used to process the current target data table by partitioning, wherein the first partitioning mode specifically refers to extracting the rows corresponding to the specific data contents and arranging all the extracted rows into one partition; If no, the second partitioning mode is used to process the current target data table by partitioning, wherein the second partitioning mode specifically refers to processing the current target data table by partitioning according to the maximum row threshold or / and the maximum column threshold.

6. The CDC data resource synchronization method based on partition mode according to claim 5, characterized in that, The step of distinguishing the preprocessing result according to the preset naming rule specifically includes: If the partitioning mode corresponding to the preprocessing result is the first partitioning mode, the table name of the corresponding data table before this partitioning processing and the specific data content at this partitioning processing are obtained, and the table field information corresponding to the table name and the specific data content are spliced as the distinguished name of the preprocessing result; If the partitioning mode corresponding to the preprocessing result is the second partitioning mode, the table name of the corresponding data table before this partitioning processing and the partitioning number at this partitioning processing are obtained, and the table name and the partitioning number are spliced as the distinguished name of the preprocessing result; The step of synchronizing the target data resource to the target end through the distinguished name specifically includes: Judging whether the table field information involved in the data resource synchronization of the target end is contained in the string corresponding to the distinguished name; If the table field information is contained in the string, the subform corresponding to the distinguished name is taken as a search field, and the cells in the subform for data resource synchronization are searched and synchronized; If the table field information is not contained in the string, it is judged whether the table field information is the table field information of the subform corresponding to the distinguished name. If the table field information is the table field information of the subform corresponding to the distinguished naming, the subform corresponding to the distinguished naming is taken as a search domain, and a cell in the subform that performs data resource synchronization is searched and synchronized. If the table field information is not the table field information of the subform corresponding to the distinguished naming, the subform is not operated.

7. A CDC data resource synchronization device based on a partition mode, characterized in that, The method comprises the following steps: A log analysis module is configured to obtain data operation record logs of a data source end, analyze the data operation record logs, and obtain an analysis result. A table field information identification module is configured to identify table field information involved in data resource synchronization at a target end according to the analysis result. A table to be synchronized searching module is configured to search all data tables to be synchronized in the target end by taking any table field information in the table field information as search information. A form preprocessing module is configured to preprocess all data tables to be synchronized according to a preset preprocessing mode, and obtain a preprocessing result, which specifically comprises the following steps: Step 401: Obtain row and column values of each data table to be synchronized. Step 402: According to a preset filtering rule and the row and column values of each data table to be synchronized, filter out data tables that need to be processed by partitioning. Step 403: According to a preset partitioning rule, partition all data tables that need to be processed by partitioning, and obtain subforms after partitioning. Step 404: Obtain row and column values of each subform, and iteratively execute steps 402 to 404 until no form needs to be processed by partitioning, and terminate iteration. Step 405: Obtain final subforms obtained by partitioning as the preprocessing result. A distinguished naming module is configured to distinguish name the preprocessing result according to a preset naming rule. A data resource synchronization module is configured to synchronize target data resources to the target end through the distinguished naming.

8. A computer device comprising a memory and a processor, wherein the memory stores computer readable instructions, and the processor executes the computer readable instructions to implement the steps of the partition mode based CDC data resource synchronization method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer readable storage medium stores computer readable instructions, and the computer readable instructions are executed by the processor to implement the steps of the partition mode based CDC data resource synchronization method according to any one of claims 1 to 6.

Images