Data processing method, device, electronic device and storage medium

Abstract

The present disclosure provides a data processing method, device, electronic device, and storage medium for use in the field of computer technology. The method comprises: obtaining a maintenance date set for a historical data table, the maintenance date set consisting of non-repeated creation dates of the data maintained in the historical data table; parsing the table structure information of the current data table to obtain fields and field information in real time, and dividing the fields into common fields, unique fields, and partition fields based on the field information and predefined field classification rules; generating SQL based on the common fields, unique fields, partition fields, and a preconfigured transcoding field mapping table; traversing each creation date in the maintenance date set, executing SQL to filter target data from the historical data table; and adding the filtered target data to the data partition corresponding to the current data table according to the creation date. This solution effectively solves the problem of manually repairing the data consistency and accuracy between the current data table and the historical table after the source system maintains the historical data table in the data warehouse.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to a data processing method, device, electronic device, and storage medium. Background Art

[0002] In various business systems, source data serves as the core source of data for data warehouses. Due to the high complexity of business scenarios and large data volumes, source systems frequently undergo manual data maintenance. In these business scenarios, stream-based data typically has a high frequency of incremental data entry. To address data storage pressures, source systems move historical data into archive tables, retaining only active data in current tables. This physically separates the current and historical repositories within the source system architecture.

[0003] Due to data extraction performance limitations, data warehouses typically store the source system's current data table (referred to as the current table) and historical data table (referred to as the historical table) incrementally according to different time dimensions: the current table is partitioned by data creation date, and the historical table is partitioned by archiving date. When the source system manually maintains archived historical data records (such as data value corrections), since the maintained records have already been migrated to the historical database, the source system only needs to operate on the historical data table. However, on the data warehouse side, the current table is partitioned by creation date and retains all historical data, requiring additional manual maintenance of the historical versions of the current table data that have already been stored. Therefore, after the source system maintains the historical data table, the data warehouse layer must manually repair data consistency issues in the current data table in addition to maintaining the historical data table.

[0004] Because the data records being maintained have been archived, extracting partition data for a specific creation date directly from the current database in the source system will miss the archived historical records. This requires manual intervention to repair the current data table, making automated processing impossible through the scheduling system. This operating mode significantly increases the operational burden of the data warehouse, and frequent manual maintenance also poses a serious challenge to ensuring consistency across data tables. Summary of the Invention

[0005] The present disclosure provides a data processing method, device, electronic device and storage medium to at least solve the above technical problems existing in the prior art.

[0006] According to a first aspect of the present disclosure, a data processing method is provided, the method comprising:

[0007] Obtaining a maintenance date set of a historical data table, wherein the maintenance date set is composed of non-repeated creation dates of the data maintained in the historical data table;

[0008] Parse the table structure information of the current data table to obtain fields and field information in real time, and divide the fields into common fields, unique fields and partition fields according to the field information and predefined field classification rules;

[0009] Generate a structured query statement based on the common fields, unique fields, partition fields, and a preconfigured transcoding field mapping table;

[0010] Traversing each creation date in the maintenance date set, executing the structured query statement to filter target data from the historical data table;

[0011] The filtered target data is added to the data partition corresponding to the current data table according to the creation date.

[0012] In one possible implementation, obtaining a maintenance date set of the historical data table includes:

[0013] Extracting the update time of the historical data table;

[0014] Perform a date query operation to filter non-repeated creation dates whose update time is equal to the specified maintenance date and whose creation date is not equal to the specified maintenance date.

[0015] In one possible implementation, the field information includes a field name, a field type, and a field comment;

[0016] The fields are divided into common fields, unique fields and partition fields according to the field information and predefined field classification rules, including:

[0017] Classify fields whose field names belong to a predefined partition identification set into partition fields, wherein the predefined partition identification set includes a data processing task identification field and a date partition identification field;

[0018] Classify fields whose names contain predefined source identifiers as unique fields;

[0019] Classify the remaining fields as public fields.

[0020] In one possible implementation, generating a structured query statement based on the common fields, unique fields, partition fields, and a preconfigured transcoding field mapping table includes:

[0021] Generate a first expression for the common field: replace the null value in the numeric field with a preset value and retain the original field type; directly reference the original field name for the non-numeric field;

[0022] Generating a second expression for the unique field: assigning a system source identifier field to a predefined system code, assigning a data table identifier field to a fixed table name, and assigning a timestamp field to the current system time;

[0023] A third expression is generated for the partition field: the date partition field is assigned a value of the first date format of the current creation date; the task identifier partition field is assigned a value of a predefined task name constant;

[0024] Identify the transcoding fields specified in the preconfigured transcoding field mapping table and generate the fourth expression: construct the standard code value table association logic, using the source system identifier and the encoding type identifier as association conditions; generate the code value matching result expression, and output a string consisting of the delimiter, the original field value, and the exception identifier when the match fails; directly reference the original field name for non-transcoding fields;

[0025] The first expression, the second expression, the third expression and the fourth expression are integrated to obtain the structured query statement.

[0026] In one embodiment, traversing each creation date in the maintenance date set and executing the structured query statement to filter target data from the historical data table includes:

[0027] Convert the creation date to the first date format and the second date format;

[0028] Inject the first date format into a structured query statement, and execute a query operation to obtain target data, including:

[0029] Selecting target data whose partition date is equal to the second date format from the current data table;

[0030] Target data that meets the following conditions is selected from the historical data table: a creation date is equal to the first date format, and a partition date is not less than the first date format.

[0031] In one possible implementation, the step of adding the filtered target data into the data partition corresponding to the current data table according to the creation date includes:

[0032] injecting the first date format and a predefined task identifier constant into the third expression;

[0033] Based on the third expression, the target data is batch-written into the data partition corresponding to the creation date of the current data table.

[0034] According to a second aspect of the present disclosure, there is provided a data processing device, the device comprising:

[0035] An acquisition module, configured to acquire a maintenance date set of a historical data table, wherein the maintenance date set is composed of non-repeated creation dates of the data maintained in the historical data table;

[0036] The parsing module is used to parse the table structure information of the current data table to obtain fields and field information in real time, and divide the fields into common fields, unique fields and partition fields according to the field information and predefined field classification rules;

[0037] A generation module, configured to generate a structured query statement based on the common fields, unique fields, partition fields, and a preconfigured transcoding field mapping table;

[0038] An execution module, configured to traverse each creation date in the maintenance date set and execute the structured query statement to filter target data from the historical data table;

[0039] The insert module is used to add the filtered target data into the data partition corresponding to the current data table according to the creation date.

[0040] In one embodiment, the acquisition module is specifically used to: extract the update time of the historical data table; perform a date query operation to filter non-repeated creation dates whose update time is equal to the specified maintenance date and whose creation date is not equal to the specified maintenance date.

[0041] According to a third aspect of the present disclosure, there is provided an electronic device, including:

[0042] at least one processor; and a memory communicatively connected to the at least one processor; wherein,

[0043] The memory stores instructions that can be executed by the at least one processor. The instructions are executed by the at least one processor to enable the at least one processor to perform the method described in the present disclosure.

[0044] According to a fourth aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause the computer to execute the method described in the present disclosure.

[0045] The data processing method, device, electronic device and storage medium disclosed in the present invention include: obtaining a maintenance date set of a historical data table, wherein the maintenance date set is composed of non-repeated creation dates of the data maintained in the historical data table; parsing the table structure information of the current data table to obtain fields and field information, and dividing the fields into common fields, unique fields and partition fields according to the field information and predefined field classification rules; generating a structured query statement based on the common fields, unique fields and partition fields and a preconfigured transcoding field mapping table; traversing each creation date in the maintenance date set, executing the structured query statement to filter target data from the historical data table; and inserting the filtered target data into the data partition corresponding to the current data table according to the creation date. This solution effectively solves the problem of manually repairing the data consistency and accuracy of the current data table and the historical data table after the source system maintains the historical data table in the data warehouse. Through the automated process, the situation of missing archived historical records due to directly extracting data from the current library of the source system is avoided, and automated processing is achieved, which significantly reduces the operation and maintenance burden of the data warehouse, while ensuring data consistency across data tables and improving data processing efficiency and accuracy.

[0046] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present disclosure, nor are they intended to limit the scope of the present disclosure. Other features of the present disclosure will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] The above and other objects, features and advantages of the exemplary embodiments of the present disclosure will become readily understood by reading the detailed description below with reference to the accompanying drawings, in which several embodiments of the present disclosure are shown by way of example and not limitation, wherein:

[0048] In the drawings, the same or corresponding reference numerals denote the same or corresponding parts.

[0049] Figure 1 The implementation process of the data processing method of the embodiment of the present disclosure is shown Figure 1 ;

[0050] Figure 2 The implementation process of the data processing method of the embodiment of the present disclosure is shown Figure 2 ;

[0051] Figure 3 The implementation process of the data processing method of the embodiment of the present disclosure is shown Figure 3 ;

[0052] Figure 4 The implementation process of the data processing method of the embodiment of the present disclosure is shown Figure 4 ;

[0053] Figure 5The implementation process of the data processing method of the embodiment of the present disclosure is shown Figure 5 ;

[0054] Figure 6 The implementation process of the data processing method of the embodiment of the present disclosure is shown Figure 6 ;

[0055] Figure 7 A schematic structural diagram of a data processing device according to an embodiment of the present disclosure is shown;

[0056] Figure 8 A schematic diagram of the structure of an electronic device according to an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0057] To make the purposes, features, and advantages of the present disclosure more apparent and understandable, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present disclosure, not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of the present disclosure without creative work shall fall within the scope of protection of the present disclosure.

[0058] The present disclosure provides a data processing method, such as Figure 1 As shown, the method includes:

[0059] Step 101: Obtain a maintenance date set of a historical data table, where the maintenance date set consists of non-repeated creation dates of maintained data in the historical data table.

[0060] In this example, when the source system maintains data in the historical data table (His table), a SQL query (SELECT DISTINCT create_time FROM His table WHERE update_time = 'maintenance date' AND create_time <> 'maintenance date') is executed to automatically identify unique creation dates that are being maintained. These dates form the maintenance date set (DATA2), which is used to locate partitions in the current table with missing data due to data archiving. For example, if data created on January 1, 2025 is maintained in May 2025, the partition on January 1, 2025, needs to be repaired. This ensures that all historical partitions that need repair are covered by eliminating manual screening.

[0061] Step 102: Parse the table structure information of the current data table to obtain fields and field information in real time, and divide the fields into common fields, unique fields, and partition fields according to the field information and predefined field classification rules.

[0062] In this example, common fields are business fields (such as amount and term) shared by both the current data layer (sa) and the historical storage layer (shm). Unique fields are technical fields (such as the system source identifier) ​​that exist only in the shm layer. Partition fields include the task identifier (ETL_JOB) and the date partition identifier (DT).

[0063] Parsing the current data table's structure involves dynamically reading the table's metadata to obtain fields and field information, and then classifying the fields according to predefined rules. Specifically, by dynamically executing DESCRIBE, the latest structure of the production table (i.e., the current data table) is obtained in real time. Fields are then classified according to predefined rules: 1) Partition fields are stored in DATA6; 2) Unique fields are stored in DATA4; 3) Common fields are stored in DATA3. This automated field classification mechanism can adapt to changes in the table structure.

[0064] Step 103: Generate a structured query statement based on the common fields, unique fields, partition fields and pre-configured transcoding field mapping table.

[0065] In this example, the process of generating a structured query statement involves constructing four types of expressions, corresponding to common fields, unique fields, partition fields, and transcoded fields generated by the preconfigured transcoding field mapping table. Ultimately, all expressions are integrated to form a complete structured query statement, SQL.

[0066] Step 104: traverse each creation date in the maintenance date set, and execute the structured query statement to filter target data from the historical data table.

[0067] In this example, data is filtered based on each creation date when traversing the maintenance date collection, and then a structured query statement is executed to filter the target data from the historical data table. In this way, the time dimension is used to ensure complete data completion.

[0068] Step 105: Add the filtered target data into the data partition corresponding to the current data table according to the creation date.

[0069] In this example, the target data is inserted using a partition overwrite operation, writing to the data partition corresponding to the creation date of the current table. By precisely locating the repair location and overwriting the original partition data, the integrity of the data operation is guaranteed and the risk of partial writes is avoided.

[0070] The present disclosure provides a data processing method, including: obtaining a maintenance date set of a historical data table, the maintenance date set consisting of non-repeated creation dates of the data maintained in the historical data table; parsing the table structure information of the current data table to obtain fields and field information, and dividing the fields into common fields, unique fields, and partition fields based on the field information and predefined field classification rules; generating a structured query statement based on the common fields, unique fields, partition fields, and a preconfigured transcoding field mapping table; traversing each creation date in the maintenance date set, executing the structured query statement to filter target data from the historical data table; and inserting the filtered target data into the data partition corresponding to the creation date of the current data table. This solution effectively solves the problem of manually repairing the data consistency and accuracy of the current data table and the historical data table after the source system maintains the historical data table in the data warehouse. Through the automated process, the situation of missing archived historical records due to directly extracting data from the current library of the source system is avoided, and automated processing is achieved, which significantly reduces the operation and maintenance burden of the data warehouse, while ensuring data consistency across data tables and improving data processing efficiency and accuracy.

[0071] In one example, the maintenance date set of the historical data table is obtained, such as Figure 2 Shown, including:

[0072] Step 201: Extract the update time of the historical data table.

[0073] In this example, the update time refers to the business date when the source system performed maintenance on the historical data table, identifying the point in time when the data was modified. The update time of the historical data table (S1G_ASSET_REPAY_SERIAL_his) is dynamically extracted using a JMP schedule parameter. Specifically, a business date parameter (in the format YYYY-MM-DD, such as 2025-05-10) is passed to the JMP foreground schedule task. This parameter directly corresponds to the actual date when the source system performed the data maintenance operation.

[0074] Step 202: Execute a date query operation to filter non-repeated creation dates whose update time is equal to the specified maintenance date and whose creation date is not equal to the specified maintenance date.

[0075] In this example, distinct creation dates refer to the list of dates deduplicated using SELECT DISTINCT create_time, preventing repeated repairs of data with the same date. The specific date filtering query involves a dual filtering mechanism: first, it filters records whose update time equals the specified maintenance date; second, it excludes records created on that day by excluding records whose creation date does not equal the specified maintenance date; and finally, it extracts the distinct creation date set (i.e., DATA2) for records that meet these conditions.

[0076] In one example, the field information includes a field name, a field type, and a field comment;

[0077] According to the field information and predefined field classification rules, the fields are divided into common fields, unique fields and partition fields, such as Figure 3 Shown, including:

[0078] Step 301: Classify fields whose field names belong to a predefined partition identification set into partition fields. The predefined partition identification set includes a data processing task identification field and a date partition identification field.

[0079] In this example, the predefined partition identifier set refers to predefined key metadata field identifiers, including: Data processing task identifier field (ETL_JOB): This uniquely identifies the data repair task. Date partition identifier field (DT): This stores the data partition date (in year-month-day format). Field names that match predefined identifiers (for example, field names like DT or ETL_JOB) are classified as partition fields and stored in DATA6.

[0080] Step 302: Classify fields whose field names contain predefined source identifiers into unique fields.

[0081] In this example, the predefined source identifier refers to a proprietary technical identifier in the historical storage layer (shm), for example: system source field (such as SYS_SRC): marks the system code of the data source.

[0082] By identifying fields whose names contain specific technical identifiers (such as prefixes / suffixes such as SRC and SYS) and are not partition fields, they are classified as unique fields and stored in DATA4.

[0083] Step 303: Classify the remaining fields into public fields.

[0084] In this example, common fields refer to business fields shared by both the current data layer (sa) and the historical storage layer (shm), such as basic business attributes like loan amount (LOAN_AMT) and repayment term (TERM). In other words, the remaining fields not categorized in steps 301 and 302 are by default grouped into the common field set and stored in DATA3.

[0085] In one example, the structured query statement is generated based on the common fields, unique fields, partition fields, and pre-configured transcoding field mapping table, such as Figure 4 Shown, including:

[0086] Step 401: Generate a first expression for the common field: replace the empty value in the numeric field with a preset value and retain the original field type; the non-numeric field directly references the original field name.

[0087] In this example, the first expression is generated for public fields. Numeric fields in the public fields require null value conversion logic to ensure data integrity. Non-numeric fields can directly reference the original field name to avoid invalid conversions. For example, a numeric field (such as loan_amount) generates a null value conversion expression: COALESCE(loan_amount, 0) AS loan_amount, preserving the original field type. Non-numeric fields (such as loan_id) can directly reference the original field name.

[0088] Step 402: Generate a second expression for the unique field: assign a system source identifier field to a predefined system code, assign a data table identifier field to a fixed table name, and assign a timestamp field to the current system time.

[0089] In this example, static assignment is performed on the technical identification field in the unique field set to generate the second expression, in which: 1) the system source identifier (such as SYS_SRC) is assigned to the predefined system code 'S1G' (marking the source of the online loan system); 2) the data table identifier is assigned to a fixed table name constant; 3) the timestamp field (such as ETL_UPD_TIME) is assigned to the current system time CURRENT_TIMESTAMP.

[0090] Step 403: Generate a third expression for the partition field: assign the date partition field a value of the first date format of the current creation date; assign the task ID partition field a value of a predefined task name constant.

[0091] In this example, fixed values ​​are assigned to the partitioning fields: 1) The date partitioning field (DT) is dynamically assigned a value based on the creation date, using the first date format (tens-digit format) (e.g., 2025-01-05); 2) The job identifier field (ETL_JOB) is assigned the predefined constant 'S1G_ASSET_REPAY_SERIAL'. By injecting dynamic date parameters into the partition key, precise partition positioning is achieved.

[0092] Step 404: Identify the transcoding field specified in the preconfigured transcoding field mapping table and generate a fourth expression: construct the standard code value table association logic, with the source system identifier and the encoding type identifier as association conditions; generate a code value matching result expression, and output a string consisting of a separator, the original field value and the exception identifier when the match fails; directly reference the original field name for the non-transcoding field.

[0093] In this example, the fields to be transcoded (such as currency and document type) are located based on the preconfigured mapping table cast_value=['currency_type','T99_CCY_CDE']. The source system identifier (SRCTAB_CD='S1G') and the encoding type identifier (CDE_TYPE='1') are used as the association conditions. If a match succeeds, the standard code value is output. If a match fails, a string consisting of the separator #, the original value, and the exception identifier 999 is generated (for example, CONCAT('#', original value,'999').

[0094] Step 405: Integrate the first expression, the second expression, the third expression and the fourth expression to obtain the structured query statement.

[0095] In this example, the pre-order expressions are integrated to generate the final SQL statement: the first expression (common field processing), the second expression (unique field assignment), the third expression (partition field assignment), and the fourth expression (transcoded field association) are combined in the execution order to form a complete structured query statement. This integrated statement includes: 1) a field selection list; 2) the logic for joining the historical data table with the code value table; and 3) a partition filter condition (WHERE create_time='$date10$').

[0096] In one example, the traversal of each creation date in the maintenance date set executes the structured query statement to filter target data from the historical data table, such as Figure 5 Shown, including:

[0097] Step 501: Convert the creation date into a first date format and a second date format.

[0098] In this example, each creation date in the maintenance date set is converted into two formats: the first date format (DATA10): the ten-digit partition format YYYY-MM-DD (such as 2025-01-05), which is used for historical data table timestamp condition filtering and data warehouse partition key assignment; the second date format (DATA8): the eight-digit date format YYYYMMDD (such as 20250105), which is used for current warehouse data table timestamp condition filtering.

[0099] Step 502: inject the first date format into a structured query statement, and execute a query operation to obtain target data.

[0100] Inject the first date format (DATA10) into the structured query statement and execute the query: Dynamic injection: Replace the partition key value in the SQL with $date10$. Then execute the operation and run the pre-built complete SQL (including field processing, code value association, and other logic) to filter missing data from the historical data table and the current data table. The specific execution steps are as follows:

[0101] Step 5021: Select target data with a partition date equal to the second date format from the current data table.

[0102] In this example, data with partition dates in the second date format (eight-digit format) is filtered from the current table. Valid data is obtained by combining transcoding field association logic (such as currency transcoding). For example, the WHERE DT='$date8$' clause limits the partition range (for example, DT='20250105').

[0103] Step 5022: Select target data that meets the following conditions from the historical data table: the creation date is equal to the first date format, and the partition date is not less than the first date format.

[0104] In this example, target data is filtered from the historical data table to meet two conditions: the creation date is equal to the first date format (ten-digit format) (for example, CREATE_TIME = '2025-01-05') and the partition date is not less than the first date format (for example, DT >= '2025-01-05'). Double timestamp verification of CREATE_TIME and DT ensures the integrity of the archived data.

[0105] In one example, the filtered target data is added to the data partition corresponding to the current data table according to the creation date, such as Figure 6 Shown, including:

[0106] Step 601: Inject the first date format and predefined task identifier constant into the third expression.

[0107] In this example, the first date format (a ten-digit standardized format, such as 2025-01-05) and a predefined task identifier constant (such as "loan_repair_task") are injected into the partition assignment expression (the third expression): the date partition field (DT) is assigned a ten-digit date value (DT = '2025-01-05'); the task identifier field (ETL_JOB) is assigned a constant task name.

[0108] Step 602: Based on the third expression, write the target data in batches into the data partition corresponding to the creation date of the current data table.

[0109] In this example, based on the injected third expression, a partition overwrite operation (INSERT OVERWRITETABLE ... PARTITION (DT='2025-06-05')) is used to write the target data in batches to the target partition of the current table. This operation directly associates the data partition with the creation date (that is, the partition key DT value corresponds to the repair date), and automatically filters out dirty data.

[0110] The present disclosure also provides a data processing device, such as Figure 7 As shown, the device includes:

[0111] An acquisition module 701 is configured to acquire a maintenance date set of a historical data table, wherein the maintenance date set is composed of non-repeated creation dates of the maintained data in the historical data table;

[0112] The parsing module 702 is used to parse the table structure information of the current data table to obtain fields and field information in real time, and classify the fields into common fields, unique fields and partition fields according to the field information and predefined field classification rules;

[0113] A generating module 703 is configured to generate a structured query statement based on the common fields, unique fields, partition fields, and a preconfigured transcoding field mapping table;

[0114] An execution module 704 is configured to traverse each creation date in the maintenance date set and execute the structured query statement to filter target data from the historical data table;

[0115] The inserting module 705 is used to add the filtered target data into the data partition corresponding to the current data table according to the creation date.

[0116] In one example, the acquisition module 701 is specifically configured to:

[0117] Extracting the update time of the historical data table;

[0118] Perform a date query operation to filter non-repeated creation dates whose update time is equal to the specified maintenance date and whose creation date is not equal to the specified maintenance date.

[0119] In one example, the field information includes a field name, a field type, and a field comment; the parsing module 702 is specifically configured to:

[0120] Classify fields whose field names belong to a predefined partition identification set into partition fields, wherein the predefined partition identification set includes a data processing task identification field and a date partition identification field;

[0121] Classify fields whose names contain predefined source identifiers as unique fields;

[0122] Classify the remaining fields as public fields.

[0123] In one example, the generating module 703 is specifically configured to:

[0124] Generate a first expression for the common field: replace the null value in the numeric field with a preset value and retain the original field type; directly reference the original field name for the non-numeric field;

[0125] Generating a second expression for the unique field: assigning a system source identifier field to a predefined system code, assigning a data table identifier field to a fixed table name, and assigning a timestamp field to the current system time;

[0126] A third expression is generated for the partition field: the date partition field is assigned a value of the first date format of the current creation date; the task identifier partition field is assigned a value of a predefined task name constant;

[0127] Identify the transcoding fields specified in the preconfigured transcoding field mapping table and generate the fourth expression: construct the standard code value table association logic, using the source system identifier and the encoding type identifier as association conditions; generate the code value matching result expression, and output a string consisting of the delimiter, the original field value, and the exception identifier when the match fails; directly reference the original field name for non-transcoding fields;

[0128] The first expression, the second expression, the third expression and the fourth expression are integrated to obtain the structured query statement.

[0129] In one example, the execution module 704 is specifically configured to:

[0130] Convert the creation date to the first date format and the second date format;

[0131] Inject the first date format into a structured query statement, and execute a query operation to obtain target data, including:

[0132] Selecting target data whose partition date is equal to the second date format from the current data table;

[0133] Target data that meets the following conditions is selected from the historical data table: a creation date is equal to the first date format, and a partition date is not less than the first date format.

[0134] In one example, the insertion module 705 is specifically configured to:

[0135] injecting the first date format and a predefined task identifier constant into the third expression;

[0136] Based on the third expression, the target data is batch-written into the data partition corresponding to the creation date of the current data table.

[0137] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device and a readable storage medium.

[0138] Figure 8 A schematic block diagram of an example electronic device 800 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are provided as examples only and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0139] like Figure 8 As shown, device 800 includes a computing unit 801, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 802 or a computer program loaded from a storage unit 808 into a random access memory (RAM) 803. RAM 803 may also store various programs and data required for the operation of device 800. Computing unit 801, ROM 802, and RAM 803 are interconnected via a bus 804. An input / output (I / O) interface 805 is also connected to bus 804.

[0140] Various components in device 800 are connected to I / O interface 805, including an input unit 806, such as a keyboard, mouse, etc.; an output unit 807, such as various types of displays, speakers, etc.; a storage unit 808, such as a magnetic disk, optical disk, etc.; and a communication unit 809, such as a network card, modem, wireless communication transceiver, etc. The communication unit 809 allows device 800 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0141] The computing unit 801 can be any general-purpose and / or specialized processing component with processing and computing capabilities. Some examples of the computing unit 801 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various specialized artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 801 performs the various methods and processes described above, such as the data processing method. For example, in some embodiments, the data processing method may be implemented as a computer software program tangibly embodied in a machine-readable medium, such as the storage unit 808. In some embodiments, part or all of the computer program may be loaded and / or installed onto the device 800 via the ROM 802 and / or the communication unit 809. When the computer program is loaded into the RAM 803 and executed by the computing unit 801, one or more steps of the data processing method described above may be performed. Alternatively, in other embodiments, the computing unit 801 may be configured to perform the data processing method by any other suitable means (e.g., via firmware).

[0142] Various embodiments of the systems and techniques described above can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system-on-chip systems (SOCs), programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system that includes at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0143] The program code for implementing the method of the present disclosure can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device so that when the program code is executed by the processor or controller, the functions / operations specified in the flow chart and / or block diagram are implemented. The program code can be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0144] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of machine-readable storage media may include an electrical connection based on one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), optical fibers, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0145] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0146] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0147] A computer system may include a client and a server. The client and server are generally remote from each other and typically interact through a communication network. The client-server relationship arises through computer programs running on the respective computers and having a client-server relationship with each other. The server may be a cloud server, a server in a distributed system, or a server integrated with a blockchain.

[0148] It should be understood that the various forms of processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this disclosure can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions of this disclosure can be achieved, and this document is not limited here.

[0149] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features being referred to. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one such feature. Throughout the present disclosure, "plurality" means two or more, unless otherwise specifically defined.

[0150] The above description is merely a specific embodiment of the present disclosure, but the scope of protection of the present disclosure is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this disclosure should be included in the scope of protection of the present disclosure. Therefore, the scope of protection of the present disclosure should be based on the scope of protection of the claims.

Claims

1. A data processing method, characterized in that: The method comprises: Obtaining a maintenance date set of a historical data table, wherein the maintenance date set is composed of non-repeated creation dates of the data maintained in the historical data table; Parse the table structure information of the current data table to obtain fields and field information in real time, and divide the fields into common fields, unique fields and partition fields according to the field information and predefined field classification rules; Generate a structured query statement based on the common fields, unique fields, partition fields, and a preconfigured transcoding field mapping table; Traversing each creation date in the maintenance date set, executing the structured query statement to filter target data from the historical data table; Fill the filtered target data into the data partition corresponding to the current data table according to the creation date; The generating of a structured query statement based on the common fields, unique fields, partition fields, and the preconfigured transcoding field mapping table includes: Generate a first expression for the common field: replace the null value in the numeric field with a preset value and retain the original field type; directly reference the original field name for the non-numeric field; Generating a second expression for the unique field: assigning a system source identifier field to a predefined system code, assigning a data table identifier field to a fixed table name, and assigning a timestamp field to the current system time; A third expression is generated for the partition field: the date partition field is assigned a value of the first date format of the current creation date; the task identifier partition field is assigned a value of a predefined task name constant; Identify the transcoding fields specified in the preconfigured transcoding field mapping table and generate the fourth expression: construct the standard code value table association logic, using the source system identifier and the encoding type identifier as association conditions; generate the code value matching result expression, and output a string consisting of the delimiter, the original field value, and the exception identifier when the match fails; directly reference the original field name for non-transcoding fields; The first expression, the second expression, the third expression and the fourth expression are integrated to obtain the structured query statement.

2. The method according to claim 1, characterized in that The step of obtaining a maintenance date set of the historical data table includes: Extracting the update time of the historical data table; Perform a date query operation to filter non-repeated creation dates whose update time is equal to the specified maintenance date and whose creation date is not equal to the specified maintenance date.

3. The method according to claim 1, characterized in that The field information includes field name, field type and field comment; The fields are divided into common fields, unique fields and partition fields according to the field information and predefined field classification rules, including: Classify fields whose field names belong to a predefined partition identification set into partition fields, wherein the predefined partition identification set includes a data processing task identification field and a date partition identification field; Classify fields whose names contain predefined source identifiers as unique fields; Classify the remaining fields as public fields.

4. The method according to claim 1, wherein The traversing each creation date in the maintenance date set and executing the structured query statement to filter target data from the historical data table includes: Convert the creation date to the first date format and the second date format; Inject the first date format into a structured query statement, and execute a query operation to obtain target data, including: Selecting target data whose partition date is equal to the second date format from the current data table; Target data that meets the following conditions is selected from the historical data table: a creation date is equal to the first date format, and a partition date is not less than the first date format.

5. The method according to claim 4, characterized in that The step of adding the filtered target data into the data partition corresponding to the current data table according to the creation date includes: injecting the first date format and a predefined task identifier constant into the third expression; Based on the third expression, the target data is batch-written into the data partition corresponding to the creation date of the current data table.

6. A data processing device, characterized in that: The device comprises: An acquisition module, configured to acquire a maintenance date set of a historical data table, wherein the maintenance date set is composed of non-repeated creation dates of the data maintained in the historical data table; The parsing module is used to parse the table structure information of the current data table to obtain fields and field information in real time, and divide the fields into common fields, unique fields and partition fields according to the field information and predefined field classification rules; A generation module, configured to generate a structured query statement based on the common fields, unique fields, partition fields, and a preconfigured transcoding field mapping table; An execution module, configured to traverse each creation date in the maintenance date set and execute the structured query statement to filter target data from the historical data table; An insert module is used to add the filtered target data into the data partition corresponding to the current data table according to the creation date; The generation module is specifically used for: Generate a first expression for the common field: replace the null value in the numeric field with a preset value and retain the original field type; directly reference the original field name for the non-numeric field; Generating a second expression for the unique field: assigning a system source identifier field to a predefined system code, assigning a data table identifier field to a fixed table name, and assigning a timestamp field to the current system time; A third expression is generated for the partition field: the date partition field is assigned a value of the first date format of the current creation date; the task identifier partition field is assigned a value of a predefined task name constant; Identify the transcoding fields specified in the preconfigured transcoding field mapping table and generate the fourth expression: construct the standard code value table association logic, using the source system identifier and the encoding type identifier as association conditions; generate the code value matching result expression, and output a string consisting of the delimiter, the original field value, and the exception identifier when the match fails; directly reference the original field name for non-transcoding fields; The first expression, the second expression, the third expression and the fourth expression are integrated to obtain the structured query statement.

7. The device according to claim 6, characterized in that The acquisition module is specifically used to: extract the update time of the historical data table; perform a date query operation, and filter non-repeated creation dates whose update time is equal to the specified maintenance date and whose creation date is not equal to the specified maintenance date.

8. An electronic device, characterized in that: include: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 5.

9. A non-transitory computer-readable storage medium storing computer instructions, characterized in that: The computer instructions are used to enable a computer to execute the method according to any one of claims 1 to 5.

Images