Data processing methods and apparatus

Abstract

This disclosure proposes a data processing method and apparatus, wherein the method includes: obtaining the data table type corresponding to a first data table to be collected; determining a matching data collection strategy based on the data table type; collecting target data in the first data table based on the data collection strategy; and updating the data in a second data table to be synchronized with the first data table based on the target data.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence technology, and in particular to a data processing method and apparatus. Background Technology

[0002] As the digital transformation of the financial industry deepens, related business systems are becoming increasingly complex, with data sources being widespread and massive in volume. To support advanced data applications such as refined operations and intelligent risk control, data platforms have been built in related technologies. As a unified hub for data asset management and service capabilities, the data acquisition platform within the data platform bears the crucial responsibility of efficiently, accurately, and stably aggregating raw data from various source systems. Therefore, how to achieve accurate data acquisition has become an important technical issue in this field. Summary of the Invention

[0003] This disclosure provides a data processing method and apparatus to at least partially solve one of the technical problems in the related art. The technical solution of this disclosure is as follows:

[0004] According to a first aspect of the present disclosure, a data processing method is provided, comprising: obtaining a data table type corresponding to a first data table to be collected; determining a matching data collection strategy based on the data table type; collecting target data in the first data table based on the data collection strategy; and updating data in a second data table to be synchronized with the first data table based on the target data.

[0005] According to a second aspect of the present disclosure, a data processing apparatus is provided, comprising: an acquisition module, configured to acquire a data table type corresponding to a first data table to be data collected; a determination module, configured to determine a matching data collection strategy based on the data table type; a collection module, configured to collect target data in the first data table based on the data collection strategy; and an update module, configured to update a second data table to be synchronized with the first data table based on the target data.

[0006] According to a third aspect of the present disclosure, an electronic device is provided, comprising: a processor and a memory communicatively connected to the processor; the memory stores computer-executable instructions; the processor executes the computer-executable instructions stored in the memory to implement a data processing method as described in the first aspect of the present disclosure.

[0007] According to a fourth aspect of the present disclosure, a computer-readable storage medium is provided, which stores computer-executable instructions that, when executed by a processor, are used to implement a data processing method as described in the first aspect of the present disclosure.

[0008] According to a fifth aspect of the present disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements a data processing method as described in the first aspect of the present disclosure.

[0009] The technical solutions provided by the embodiments of this disclosure have at least the following beneficial effects: The process involves: identifying the data table type of the first data table to be collected; determining the matching data collection strategy based on the data table type; collecting target data from the first data table based on the data collection strategy; and updating the second data table to be synchronized with the first data table based on the target data. By first identifying the data table type and then matching the corresponding data collection strategy, the process adaptively selects the data collection method based on the data table's structural characteristics, avoiding inefficiencies or inaccurate data caused by using a uniform collection logic. Updating the second data table based on the collected target data enables automated and accurate data synchronization between the two data tables, improving the versatility and adaptability of data synchronization.

[0010] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0011] The above and / or additional aspects and advantages of this disclosure will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, in which: Figure 1 A schematic flowchart illustrating a data processing method provided in the first embodiment of this disclosure; Figure 2 A schematic flowchart illustrating another data processing method provided in the first embodiment of this disclosure; Figure 3 A schematic flowchart illustrating another data processing method provided in the first embodiment of this disclosure; Figure 4 A schematic flowchart illustrating another data processing method provided in the first embodiment of this disclosure; Figure 5 A schematic flowchart illustrating another data processing method provided in the first embodiment of this disclosure; Figure 6 A schematic flowchart illustrating another data processing method provided in the first embodiment of this disclosure; Figure 7 This is a schematic diagram of the structure of a data processing apparatus provided in the second embodiment of the present disclosure; Figure 8 This is a block diagram of the electronic device provided in the third embodiment of this disclosure. Detailed Implementation

[0012] To enable those skilled in the art to better understand the technical solutions of this disclosure, the technical solutions in the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings.

[0013] It should be noted that the terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this disclosure described herein can be implemented in orders other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0014] It should be noted that the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved in the technical solution disclosed herein are all carried out with the consent of the user, and all comply with the provisions of relevant laws and regulations, and do not violate public order and good morals.

[0015] In related technologies, data is mainly collected in batches through the following two methods: 1. Incremental identification method based on timestamp or version number fields: This method requires the source system data table to contain a field that identifies the record update time or an incrementing version number field. During data collection tasks, incremental data is obtained by querying records whose field is greater than the previous collection deadline or version number.

[0016] 2. Difference detection method based on full comparison: For old or closed systems that cannot provide a valid timestamp field, the strategy of "full pull + local comparison" is adopted. That is, the source table data is fully exported each time it is collected and compared with the historical snapshots stored in the data platform to identify newly added, modified or deleted records.

[0017] However, the aforementioned existing technology has the following problems: 1. High invasiveness or dependency on the source system: Timestamp schemes require the source system to maintain an accurate, consistent, and non-rollbackable update time field. However, in actual business systems, many core transaction systems, especially those based on mainframes or traditional databases, do not have such a field designed due to performance or architectural limitations; even if it exists, timestamps may be unreliable due to system clock drift, batch correction jobs, cross-time zone processing, etc., resulting in missed or repeated data collection.

[0018] 2. Inability to effectively capture physical deletion operations: Timestamp-based solutions cannot detect physical deletion operations of data. Deleted data cannot be obtained through field filtering, which leads to inconsistencies between the data in the data platform and the source system, seriously affecting data accuracy.

[0019] 3. Full comparison consumes huge resources and has poor scalability: Although full comparison can identify deletion operations, it requires transmitting and processing the entire table data each time, resulting in extremely high overhead in network bandwidth, storage input / output (I / O), and computing resources. When dealing with tables containing hundreds of millions or billions of records, such as transaction logs, the collection window cannot meet the timeliness requirements and puts performance pressure on the source system database, violating the "low-intrusion" principle.

[0020] 4. Lack of unified incremental strategy management and fault tolerance mechanism: Existing solutions usually hardcode the acquisition logic for a single table and lack the ability to adapt automated incremental strategies to different source systems and different table structures.

[0021] To address the aforementioned problems, this disclosure proposes a data processing method and apparatus. The data processing method and apparatus of embodiments of this disclosure are described below with reference to the accompanying drawings.

[0022] It should be noted that the data processing method in this embodiment is executed by a data processing device, which can be implemented by software and / or hardware and can be configured in an electronic device.

[0023] Figure 1 This is a flowchart illustrating a data processing method provided in the first embodiment of this disclosure.

[0024] like Figure 1 As shown, the data processing method includes the following steps: Step 101: Obtain the data table type corresponding to the first data table to be collected.

[0025] The first data table to be collected can refer to any data table in the source system; the source system can refer to financial business systems such as banking systems, credit systems, payment systems, and channel systems; the first data table can refer to data tables in non-real-time business scenarios such as T+1 reports, historical analysis tables, and regulatory reporting forms.

[0026] The data table type can be determined based on the business type of the business associated with the first data table, or it can be determined based on the fields included in the first data table.

[0027] In some alternative embodiments, such as Figure 2 As shown, step 101 may include the following steps: Step 1011: Obtain the metadata of the first data table; Step 1012: Based on the metadata, perform primary key field and target timestamp field detection on the first data table to obtain field detection results; wherein, the target timestamp field refers to the timestamp field whose null value rate and monotonicity meet the set requirements; the field detection results are used to indicate whether a primary key field exists in the first data table, or to indicate whether a primary key field and a target timestamp field exist in the first data table; Step 1013: Obtain the data table type based on the field detection results.

[0028] The metadata of the first data table may include basic structure metadata, data distribution metadata, data characteristics and quality metadata, etc. For example, the basic structure metadata may include database type identifier, table name, storage engine type, primary key definition, and field list and data type, etc.; the data distribution metadata may include total number of rows at the table level, average daily growth, and storage skew of partitioned tables, etc.; the data characteristics and quality metadata may include field precision, null value rate, and monotonicity, etc.

[0029] The null value rate and monotonicity meeting the set requirements can mean that the null value rate is less than the first set threshold and the field value of the timestamp field is monotonically increasing; or it can mean that the non-null value rate is greater than the second set threshold and the field value of the timestamp field is monotonically increasing.

[0030] The data table type varies depending on the field detection results. For example, if the field detection result indicates the presence of a primary key field and a target timestamp field, the data table type can be type one; if the field detection result indicates the presence of a primary key field but the absence of a target timestamp field, the data table type can be type two; and if the field detection result indicates the absence of a primary key field, the data table type can be type three. Type one, type two, and type three represent three different data table types.

[0031] This allows for the pre-definition of field-type mapping relationships between different field detection results and corresponding data table types. After obtaining the field detection results of the first data table, the matching data table type can be queried from the pre-defined field-type mapping relationship. Alternatively, the field detection results can be directly determined as the data table type.

[0032] Therefore, by detecting primary key fields and target timestamp fields that meet the requirements of null value rate and monotonicity in metadata, data table types can be objectively and standardizedly classified. Determining the data table type based on the field detection results can make data table classification more accurate, provide a reliable basis for subsequent selection of data collection strategies, and improve the intelligence level of the overall data synchronization process.

[0033] Step 102: Determine the matching data collection strategy based on the data table type.

[0034] Different data table types require different data collection strategies.

[0035] This allows for the pre-definition of type-policy mapping relationships between different data table types and corresponding data acquisition strategies. After obtaining the data table type of the first data table, a matching data acquisition strategy can be queried from the pre-defined type-policy mapping relationship.

[0036] Step 103: Based on the data acquisition strategy, collect target data from the first data table.

[0037] The target data refers to the data in the first data table that has changed relative to the data content at the set time point, such as data added, deleted, or modified after the set time point. For example, the data content in the first data table at the set time point could refer to the data content in the first data table at the previous data collection deadline.

[0038] In some alternative embodiments, such as Figure 3 As shown, step 103 may include the following steps: Step 10311, in response to the field detection result indicating that a primary key field and a target timestamp field exist in the first data table, based on the target timestamp field, incremental data within a specified time range in the first data table is collected, and a first primary key set in the first data table is obtained; Step 10312, target data is obtained based on the incremental data and the first primary key set.

[0039] Among them, the data table type corresponding to the first data table is the first type, that is, it has a primary key + a target timestamp field, and the data collection strategy can be the timestamp incremental collection + primary key set collection strategy.

[0040] The specified time range refers to the time range during which data will be collected. For example, the specified time range could be from the last data collection deadline to the current time.

[0041] Incremental data refers to newly added or modified data within a specified time range; the first primary key set includes all primary keys in the first data table to be collected. It can be understood that the first primary key set can also be called a primary key snapshot; the target data includes incremental data and the first primary key set.

[0042] Therefore, when the first data table contains both a primary key and a target timestamp field, directly collecting incremental data within a specified range based on the timestamp can ensure the collection of new and changed data while avoiding full table scans, thus reducing the amount of data collected and computational overhead. In addition, combining the target data with the first primary key set can capture deletion operations and ensure the consistency of data synchronization.

[0043] In some alternative embodiments, such as Figure 4 As shown, step 103 may further include the following steps: Step 10321, in response to the field detection result indicating that a primary key field exists in the first data table but a target timestamp field does not exist, the first data table is divided into blocks according to the set block division rules to obtain multiple first blocks, and the second data table is divided into blocks to obtain multiple second blocks; Step 10322, content consistency detection is performed on the first and second blocks that match the position; Step 10323, the target data is obtained based on the data in the first blocks that fail the content consistency detection.

[0044] Among them, the data table type corresponding to the first data table is the second type, that is, there is a primary key + there is no target timestamp field, and the data collection strategy can be a block comparison collection strategy.

[0045] The block setting rules can indicate the block setting method and block size, etc. For example, the block setting rules can indicate that every N rows in the first data table and the second data table are divided into blocks, where N is a positive integer.

[0046] The content consistency check is used to detect whether the data content in the first and second blocks that match the location is completely consistent. If the content consistency check fails, it means that the data in the first block is not completely the same as the data in the second block. That is, it means that some or all of the data in the first block has been deleted, modified, or new data has been added. Therefore, the data in the first block that failed the content consistency check can be used as the target data.

[0047] To improve the efficiency and accuracy of content consistency detection, for the first and second blocks with matching positions, a first hash value corresponding to the first block and a second hash value corresponding to the second block can be calculated. The first hash value and the second hash value are compared to determine whether the first and second blocks pass the content consistency detection. For example, the first hash value and the second hash value can be calculated based on the MD5 message digest algorithm.

[0048] Therefore, for data tables with primary keys but no target timestamp field, a block comparison method is adopted to break down the full table comparison into fine-grained block detection, reducing the pressure of single data processing; in addition, only the difference data of the blocks with inconsistent content is collected, avoiding the waste of resources caused by full collection and improving the data collection efficiency in scenarios without timestamp fields.

[0049] In some alternative embodiments, such as Figure 5 As shown, step 103 may also include the following steps: step 10331, in response to the field detection result indicating that a primary key field exists in the first data table and a target timestamp field does not exist, obtain the first primary key set in the first data table; step 10332, based on the first primary key set, obtain the target data.

[0050] Among them, the data table type corresponding to the first data table is the second type, that is, the primary key exists and the target timestamp field does not exist. The data collection strategy can also be the full primary key collection strategy.

[0051] In response to the existence of a primary key and the absence of a target timestamp field, in order to simplify the data collection logic and reduce unnecessary content comparisons, the full primary key of the first data table can be collected to obtain the first primary key set.

[0052] Therefore, for scenarios with a primary key but no target timestamp field, the primary key set can be directly collected, and the target data can be obtained based on the primary key set, which simplifies the data collection logic and reduces unnecessary content comparison.

[0053] In some alternative embodiments, such as Figure 6 As shown, step 103 may also include the following steps: step 10341, in response to the field detection result indicating that there is no primary key field in the first data table, the target data is obtained based on the data content in the first data table.

[0054] Among them, the data table type corresponding to the first data table is the third type, that is, there is no primary key, and the data collection strategy can be the full data collection strategy.

[0055] In the absence of a primary key field, the entire data of the first data table can be collected, that is, all data content in the first data table can be collected to obtain the target data.

[0056] Therefore, for data tables that do not have a primary key field, data can be collected directly based on the data content, breaking through the limitation of missing primary keys on data synchronization. Data collection and updating can be completed without relying on primary keys or timestamp fields, expanding the applicability of data synchronization methods and improving compatibility with heterogeneous data tables.

[0057] In some optional embodiments, step 103 may include the following steps: obtaining an abstract syntax tree template corresponding to the data acquisition strategy, wherein the abstract syntax tree template is used to indicate the data acquisition logic corresponding to the data acquisition strategy; generating target code matching the database language corresponding to the first data table through an adapter based on the abstract syntax tree template; and acquiring target data in the first data table based on the target code.

[0058] Each data acquisition strategy has a corresponding abstract syntax tree template, which abstracts the corresponding acquisition logic into a general intermediate state, thus decoupling the data acquisition logic from the specific database type.

[0059] The adapter translates the data acquisition logic into target code that matches the database language corresponding to the first data table. For example, the target code can be a Structured Query Language (SQL) statement.

[0060] Therefore, by uniformly encapsulating the data acquisition logic of different data acquisition strategies through an abstract syntax tree template, and then generating language code adapted to the corresponding database through an adapter, the data acquisition logic is decoupled from the specific database type. This eliminates the need to write acquisition logic separately for different databases, reduces development and maintenance costs, and improves the universality and scalability of cross-database data acquisition.

[0061] It should be noted that, in order to avoid data omissions caused by uncommitted source transactions or clock asynchronization, the generated target code can automatically push forward a safety window by the lower bound of the specified time range. That is, the specified time range is expanded to obtain a new time range, in which the new time range includes the specified time range.

[0062] It should be noted that, to avoid significant I / O overhead due to table lookups, if the first data table is stored in a database such as TDSQL or Oracle, index hints can be forcibly added to the generated target code to ensure that the query utilizes a covering index. The index hints indicate the storage address of the data content in the first data table.

[0063] Step 104: Based on the target data, update the data in the second data table that needs to be synchronized with the first data table.

[0064] The second data table can refer to a data table in the data platform used to synchronize with the first data table. For example, the second data table is a backup data table of the first data table.

[0065] In some alternative embodiments, corresponding to Figure 3In response to the field detection result indicating the existence of a primary key field and a target timestamp field in the first data table, step 104 may include the following steps: performing data addition and / or data modification operations in the second data table based on incremental data; comparing the first primary key set with the second primary key set in the second data table to obtain the target primary key; wherein, the target primary key refers to a primary key that is included in the second primary key set but not included in the first primary key set; and performing data deletion operations in the second data table based on the target primary key.

[0066] Based on the collected incremental data, data addition operations can be performed on newly added data records and data modification operations can be performed on modified data records in the second data table to achieve the synchronization of incremental data.

[0067] The target primary key is the primary key corresponding to the data deleted within a specified time range, used to indicate the data deleted from the first data table.

[0068] Therefore, by using incremental data to complete the addition and modification of data, and by identifying primary keys that no longer exist in the first data table through primary key comparison and performing deletion, full coverage of add, delete and modify operations in incremental synchronization scenarios is achieved; data consistency updates can be completed without full comparison, thus improving data synchronization efficiency.

[0069] In some alternative embodiments, corresponding to Figure 4 and Figure 5 In response to the field detection result indicating that a primary key field exists in the first data table but a target timestamp field does not exist, step 104 may include the following steps: based on the target data, obtain first data content, second data content, and third data content; wherein, the first data content is data content included in the first data table but not included in the second data table, the second data content is data content included in both the second and first data tables, and the third data content refers to data content in the first and second data tables where the primary key field is the same, but the associated content of the primary key field is different; based on the first data content, perform a data addition operation in the second data table, perform a data deletion operation in the second data table based on the second data content, and perform a data modification operation in the second data table based on the third data content.

[0070] The first data content consists of data to be added to the second data table, the second data content consists of data to be deleted from the second data table, and the third data content consists of data to be modified in the second data table.

[0071] Specifically, in response to a data collection strategy that is a block comparison collection strategy, the first block and the second block that failed the content consistency detection can be compared to obtain at least one of the first data content, the second data content, and the third data content; in response to a data collection strategy that is a full primary key collection strategy, the first primary key set can be compared with the second primary key set in the second data table to obtain at least one of the first data content, the second data content, and the third data content.

[0072] Therefore, by distinguishing between newly added data, redundant data, and data with the same primary key but inconsistent content, and performing add, delete, and modify operations respectively, accurate differentiated data updates are achieved. In addition, targeted handling of various data differences can avoid invalid updates and duplicate operations, and can still ensure the accuracy and integrity of data synchronization even without a target timestamp field.

[0073] In some alternative embodiments, corresponding to Figure 6 In response to the field detection result indicating that there is no primary key field in the first data table, step 104 may include the following steps: comparing the first data with the second data table to obtain at least one of the first data content, the second data content, and the third data content; and performing data addition, data deletion, and data modification operations in the second data table based on the first data content, the second data content, and the third data content; or, the second data table may be directly replaced by the first data table.

[0074] It should be noted that, corresponding to Figure 6 Alternatively, a pseudo-incremental strategy based on row identifier (RowID) can be adopted. That is, based on the Row ID, incremental data within a specified time range in the first data table is collected to obtain the target data; the second data table is then updated based on the target data.

[0075] In summary, the process involves: identifying the data table type of the first data table to be collected; determining the matching data collection strategy based on the data table type; collecting target data from the first data table based on the data collection strategy; and updating the second data table to be synchronized with the first data table based on the target data. By first identifying the data table type and then matching the corresponding data collection strategy, the process adaptively selects the data collection method based on the data table's structural characteristics, avoiding inefficiencies or inaccurate data caused by using a uniform collection logic. Updating the second data table based on the collected target data enables automated and accurate data synchronization between the two data tables, improving the versatility and adaptability of the data synchronization.

[0076] To implement the above embodiments, this disclosure also proposes a data processing apparatus.

[0077] Figure 7 This is a schematic diagram of the structure of a data processing apparatus provided in the second embodiment of the present disclosure.

[0078] like Figure 7 As shown, the data processing device 700 includes: an acquisition module 701, a determination module 702, a collection module 703, and an update module 704.

[0079] The acquisition module 701 is used to acquire the data table type corresponding to the first data table to be acquired; the determination module 702 is used to determine the matching data acquisition strategy according to the data table type; the acquisition module 703 is used to acquire target data in the first data table based on the data acquisition strategy; and the update module 704 is used to update the second data table to be synchronized with the first data table based on the target data.

[0080] As one possible implementation, the acquisition module 701 is specifically used to: acquire the metadata of the first data table; based on the metadata, perform primary key field and target timestamp field detection on the first data table to obtain field detection results; wherein, the target timestamp field refers to a timestamp field whose null value rate and monotonicity meet the set requirements; the field detection results are used to indicate whether a primary key field exists in the first data table, or to indicate whether a primary key field and a target timestamp field exist in the first data table; and obtain the data table type based on the field detection results.

[0081] As one possible implementation, the acquisition module 703 is specifically used to: respond to the field detection result indicating that a primary key field and a target timestamp field exist in the first data table; based on the target timestamp field, acquire incremental data in the first data table within a specified time range, and obtain the first primary key set in the first data table; and obtain the target data based on the incremental data and the first primary key set.

[0082] As one possible implementation, the update module 704 is specifically used for: performing data addition and / or data modification operations in the second data table based on incremental data; comparing the first primary key set with the second primary key set in the second data table to obtain the target primary key; wherein, the target primary key refers to a primary key that is included in the second primary key set but not included in the first primary key set; and performing data deletion operations in the second data table based on the target primary key.

[0083] As one possible implementation, the acquisition module 703 is specifically used to: respond to a field detection result indicating that a primary key field exists in the first data table but a target timestamp field does not exist, divide the first data table into blocks according to the set block division rules to obtain multiple first blocks, and divide the second data table into blocks to obtain multiple second blocks; perform content consistency detection on the first and second blocks that match the position; and obtain the target data based on the data in the first blocks that fail the content consistency detection.

[0084] As one possible implementation, the update module 704 is specifically used to: obtain first data content, second data content, and third data content based on the target data; wherein, the first data content refers to data content included in the first data table but not included in the second data table, the second data content refers to data content included in both the second and first data tables, and the third data content refers to data content in the first and second data tables that have the same primary key field but different associated content of the primary key field; based on the first data content, perform data addition operations, based on the second data content, perform data deletion operations in the second data table, and based on the third data content, perform data modification operations in the second data table.

[0085] As one possible implementation, the acquisition module 703 is specifically used to: in response to a field detection result indicating that a primary key field exists in the first data table but a target timestamp field does not exist, obtain the first primary key set in the first data table; and obtain the target data based on the first primary key set.

[0086] As one possible implementation, the acquisition module 703 is specifically used to: in response to a field detection result indicating that a primary key field does not exist in the first data table, obtain the target data based on the data content in the first data table.

[0087] As one possible implementation, the acquisition module 703 is specifically used to: obtain the abstract syntax tree template corresponding to the data acquisition strategy, wherein the abstract syntax tree template is used to indicate the data acquisition logic corresponding to the data acquisition strategy; based on the abstract syntax tree template, generate target code that matches the database language corresponding to the first data table through an adapter; and acquire target data in the first data table based on the target code.

[0088] It should be noted that the foregoing explanation of the data processing method embodiments also applies to the data processing apparatus of this embodiment, and will not be repeated here.

[0089] In summary, the process involves: identifying the data table type of the first data table to be collected; determining the matching data collection strategy based on the data table type; collecting target data from the first data table based on the data collection strategy; and updating the second data table to be synchronized with the first data table based on the target data. By first identifying the data table type and then matching the corresponding data collection strategy, the process adaptively selects the data collection method based on the data table's structural characteristics, avoiding inefficiencies or inaccurate data caused by using a uniform collection logic. Updating the second data table based on the collected target data enables automated and accurate data synchronization between the two data tables, improving the versatility and adaptability of the data synchronization.

[0090] Figure 8 This is a schematic diagram of the structure of an electronic device provided according to a third embodiment of the present disclosure. The electronic device 800 in this embodiment is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0091] like Figure 8 As shown, the above-mentioned electronic device 800 includes: The memory 801 and the processor 802 are connected by a bus 803, which connects the different components (including the memory 801 and the processor 802). The memory 801 stores a computer program, and when the processor 802 executes the program, it implements the data processing method of the present disclosure embodiment.

[0092] Bus 803 represents one or more of several bus architectures, including a memory bus or memory controller, a peripheral bus, a graphics acceleration port, a processor, or a local bus using any of the various bus architectures. For example, these architectures include, but are not limited to, the Industry Standard Architecture (ISA) bus, the Micro Channel Architecture (MAC) bus, the Enhanced ISA bus, the Video Electronics Standards Association (VESA) local bus, and the Peripheral Component Interconnect (PCI) bus.

[0093] Electronic device 800 typically includes a variety of electronic device readable media. These media can be any available media that can be accessed by electronic device 800, including volatile and non-volatile media, removable and non-removable media.

[0094] Memory 801 may also include computer system readable media in the form of volatile memory, such as random access memory (RAM) 804 and / or cache memory 805. Electronic device 800 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, storage system 806 may be used to read and write non-removable, non-volatile magnetic media (… Figure 8 Not shown; usually referred to as a "hard drive"). Although Figure 8 As not shown, a disk drive for reading and writing to a removable non-volatile disk (e.g., a "floppy disk") and an optical disk drive for reading and writing to a removable non-volatile optical disk (e.g., a CD-ROM, DVD-ROM, or other optical media) may be provided. In these cases, each drive may be connected to bus 803 via one or more data media interfaces. Memory 801 may include at least one program product having a set (e.g., at least one) of program modules configured to perform the functions of the embodiments of this disclosure.

[0095] A program / utility 808 having a set (at least one) of program modules 807 may be stored, for example, in memory 801. Such program modules 807 include, but are not limited to, an operating system, one or more application programs, other program modules, and program data. Each or some combination of these examples may include an implementation of a network environment. Program modules 807 typically perform the functions and / or methods described in the embodiments of this disclosure.

[0096] Electronic device 800 can also communicate with one or more external devices 809 (e.g., keyboard, pointing device, display 811, etc.), and with one or more devices that enable a user to interact with the electronic device 800, and / or with any device that enables the electronic device 800 to communicate with one or more other computing devices (e.g., network card, modem, etc.). This communication can be performed through input / output (I / O) interface 812. Furthermore, electronic device 800 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 813. Figure 8 As shown, network adapter 813 communicates with other modules of electronic device 800 via bus 803. It should be understood that, although not shown in the figure, other hardware and / or software modules can be used in conjunction with electronic device 800, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0097] The processor 802 executes various functional applications and data processing by running programs stored in the memory 801.

[0098] It should be noted that the implementation process and technical principles of the electronic device in this embodiment are explained in the foregoing description of the data processing method of the present disclosure embodiment, and will not be repeated here.

[0099] To implement the above embodiments, this disclosure also proposes a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the methods provided in the foregoing embodiments. The computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, or optical data storage device, etc.

[0100] To implement the above embodiments, this disclosure also proposes a computer program product, including a computer program that, when executed by a processor, implements the methods provided in the foregoing embodiments.

[0101] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the following claims.

[0102] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. A data processing method, characterized in that, Includes the following steps: Obtain the data table type corresponding to the first data table to be collected; Based on the data table type, determine the matching data collection strategy; Based on the data acquisition strategy, target data is collected from the first data table; Based on the target data, the second data table to be synchronized with the first data table is updated.

2. The method according to claim 1, characterized in that, The step of obtaining the data table type corresponding to the first data table to be collected includes: Obtain the metadata of the first data table; Based on the metadata, the first data table is tested for primary key field and target timestamp field to obtain field test results; wherein, the target timestamp field refers to a timestamp field whose null value rate and monotonicity meet the set requirements; the field test results are used to indicate whether the primary key field exists in the first data table, or to indicate whether the primary key field and target timestamp field exist in the first data table; The data table type is obtained based on the field detection results.

3. The method according to claim 2, characterized in that, The step of collecting target data in the first data table based on the data collection strategy includes: In response to the field detection result indicating the existence of the primary key field and the target timestamp field in the first data table, based on the target timestamp field, incremental data within a specified time range in the first data table is collected, and the first primary key set in the first data table is obtained; The target data is obtained based on the incremental data and the first primary key set.

4. The method according to claim 3, characterized in that, The step of updating the second data table, which is to be synchronized with the first data table, based on the target data includes: Based on the incremental data, perform data addition and / or data modification operations in the second data table; The first primary key set is compared with the second primary key set in the second data table to obtain the target primary key; wherein, the target primary key refers to a primary key that is included in the second primary key set but not included in the first primary key set; Based on the target primary key, perform a data deletion operation in the second data table.

5. The method according to claim 2, characterized in that, The step of collecting target data in the first data table based on the data collection strategy includes: In response to the field detection result indicating that the primary key field exists in the first data table but the target timestamp field does not exist, the first data table is divided into multiple first blocks according to the set block division rules, and the second data table is divided into multiple second blocks respectively. Perform content consistency checks on the first and second blocks that match the location; The target data is obtained based on the data in the first block that failed the content consistency check.

6. The method according to claim 2, characterized in that, The step of collecting target data in the first data table based on the data collection strategy includes: In response to the field detection result indicating that the primary key field exists in the first data table but the target timestamp field does not exist, the first primary key set in the first data table is obtained; The target data is obtained based on the first primary key set.

7. The method according to claim 5 or 6, characterized in that, The step of updating the second data table, which is to be synchronized with the first data table, based on the target data includes: Based on the target data, obtain first data content, second data content, and third data content; wherein, the first data content refers to data content included in the first data table but not included in the second data table, the second data content refers to data content included in both the second data table and the first data table, and the third data content refers to data content in the first data table and the second data table that have the same primary key field but different associated content of the primary key field; Based on the first data content, a data addition operation is performed in the second data table; a data deletion operation is performed in the second data table based on the second data content; and a data modification operation is performed in the second data table based on the third data content.

8. The method according to claim 2, characterized in that, The step of collecting target data in the first data table based on the data collection strategy includes: In response to the field detection result indicating that the primary key field does not exist in the first data table, the target data is obtained based on the data content of the first data table.

9. The method according to claim 1, characterized in that, The step of collecting target data in the first data table based on the data collection strategy includes: Obtain the abstract syntax tree template corresponding to the data acquisition strategy, wherein the abstract syntax tree template is used to indicate the data acquisition logic corresponding to the data acquisition strategy; Based on the abstract syntax tree template, target code matching the database language corresponding to the first data table is generated through an adapter; Based on the target code, the target data is collected from the first data table.

10. A data processing apparatus, characterized in that, include: The acquisition module is used to obtain the data table type corresponding to the first data table to be collected. The determination module is used to determine the matching data collection strategy based on the data table type; The data acquisition module is used to acquire target data from the first data table based on the data acquisition strategy. The update module is used to update the data of the second data table that needs to be synchronized with the first data table based on the target data.

Images