A data conversion method, system and apparatus

Abstract

The embodiment of the application discloses a data conversion method, system and device, the method comprises the following steps: when a trigger condition corresponding to a scheduling task is met, triggering the scheduling task, and transmitting task parameters to a push task main program; through the push task main program, target data in a database is queried based on the task parameters in a stream mode, the target data is parsed according to a template file, and the target data is written into a landing shard data file; after the target data in the database is queried and written into the landing shard data file, the total number of the target data under each data mode is determined, a text file corresponding to each data mode is generated according to the total number of the target data under each data mode and each landing shard data file under each data mode; and the text file corresponding to each data mode is compressed, and the compressed text file is uploaded to a target server. The method can ensure the timeliness and accuracy of data pushing, and realizes decoupling of the database and a data consumer.

Description

Technical Field

[0001] This application relates to the field of computer technology, specifically to a data conversion method, system, and apparatus. Background Technology

[0002] In recent years, the civil aviation industry has developed rapidly, with fare services becoming increasingly diversified and multi-dimensional, and fare data growing exponentially. How to accurately, quickly, and stably deliver fare data to data consumers, and provide them with a safe and convenient user experience, has become a major concern.

[0003] Currently, the fare calculation and fare search services still use an architecture design from over a decade ago. Data is directly queried from the database and then transferred to an in-memory database, with a cache of the previous hour's data retrieved every hour. Throughout this process, the timeliness and accuracy of the data can no longer meet the needs of current business development and product iteration. Furthermore, in the existing data transformation architecture, the database and the data consumer are tightly coupled, making maintenance extremely difficult. Summary of the Invention

[0004] This application provides a data conversion method, system, and apparatus that can ensure the timeliness and accuracy of data push and decouple the database from the data consumer.

[0005] In view of the above, the first aspect of this application provides a data conversion method, the method comprising:

[0006] When the triggering conditions corresponding to the scheduling task are met, the scheduling task is triggered, and the task parameters are transmitted to the main program of the push task.

[0007] The main program for the push task queries the target data in the database in streaming mode based on the task parameters, parses the target data according to the template file, and writes the target data into the local segmented data file; the template file is generated in advance according to the structure of the data in the database.

[0008] After querying the target data in the database and writing it into the landing shard data file, the total number of target data for each data mode queried is determined; for each data mode, a text file corresponding to the data mode is generated based on the total number of target data for the data mode queried and the landing shard data files for each data mode.

[0009] Compress the text file corresponding to each of the data patterns, and upload the compressed text file to the target server; the target server is used to support data storage and data access.

[0010] Optionally, the step of querying target data in the database using a streaming mode based on the task parameters through the main push task program, parsing the target data according to the template file, and writing the target data into the landing segmented data file includes:

[0011] The main program for the push task configures processing threads and database connections based on the task parameters and hardware resources. The number of processing threads is the maximum number of processing threads that the hardware resources can support, and the number of database connections is the maximum number of database connections that the hardware resources can support.

[0012] Through the database connection and the processing thread, the target data in the database is queried concurrently in streaming mode, and the target data is parsed according to the template file and written into the local sharded data file.

[0013] Optionally, the triggering conditions corresponding to the scheduling task include at least one of triggering time, triggering parameters, and triggering node;

[0014] The scheduling task is used to indicate either a full data push mode or an incremental data push mode; the full data push mode is used to indicate that the current data in the database is pushed to the main program of the push task, and the incremental data push mode is used to indicate that the data in the database that has been adjusted within a target time period is pushed to the main program of the push task, and the target time period is the time period between the time when the scheduling task is triggered this time and the time when the scheduling task was triggered last time.

[0015] Optionally, when multiple scheduling tasks are executed concurrently, the compressed text file generated by the main program of the push task for each scheduling task is used as a candidate compressed file; the method further includes:

[0016] After the main program of the push task generates candidate compressed files corresponding to each of the scheduling tasks and uploads the candidate compressed files corresponding to each of the scheduling tasks to the target server, the file format of each candidate compressed file is adjusted, and the push task is confirmed to be completed.

[0017] Optionally, the template file is generated in the following way:

[0018] The template file is generated by reverse engineering the code using the template and based on the data tables corresponding to each airline in the database. The template file is used to interpret the data structure in the database in text form and will be sent to the data consumer.

[0019] Optionally, for each data pattern, generating a text file corresponding to the data pattern based on the total number of target data under the queried data pattern and the data shards of each data pattern, includes:

[0020] For each of the data patterns, a shell script is invoked to merge the various data fragments in the data pattern into a file corresponding to the data pattern, and the total number of target data in the data pattern is written to the top of the text file.

[0021] Optionally, each data record in the database is configured with a push identifier field; the push identifier field is used to indicate the push status corresponding to the data record.

[0022] Optionally, the database also includes a deletion backup table corresponding to the data record table; the deletion backup table is used to record the data records deleted from its corresponding data record table.

[0023] A second aspect of this application provides a data conversion system, the system comprising:

[0024] A template generation component is used to generate template files based on the structure of data in the database and send the target files to the data consumer.

[0025] The scheduling task component is used to detect whether the triggering conditions corresponding to the scheduling task are met, and when it is determined that the triggering conditions corresponding to the scheduling task are met, the scheduling task is triggered and the task parameters are transmitted to the push task main program.

[0026] The push notification component is used to query target data in the database in a streaming mode based on the task parameters through the main push task program, parse the target data according to the template file, and write the target data into the landing shard data file; after querying the target data in the database and writing it into the landing shard data file, determine the total number of target data under each data mode; for each data mode, generate a text file corresponding to the data mode based on the total number of target data under the data mode and the landing shard data files under the data mode; compress the text file corresponding to each data mode, and upload the compressed text file to the target server; the target server is used to support data storage and data access.

[0027] A third aspect of this application provides a data conversion apparatus, the apparatus comprising:

[0028] The scheduling task triggering module is used to trigger the scheduling task when the triggering conditions corresponding to the scheduling task are met, and transmit the task parameters to the push task main program;

[0029] The data parsing module is used to query the target data in the database in streaming mode based on the task parameters through the main program of the push task, parse the target data according to the template file, and write the target data into the landing segmented data file; the template file is generated in advance according to the structure of the data in the database.

[0030] The file generation module is used to query the target data in the database and write it into the landing shard data file, and then determine the total number of target data for each data mode queried; for each data mode, based on the total number of target data for the data mode queried and the landing shard data files for each data mode, a text file corresponding to the data mode is generated.

[0031] The file upload module is used to compress the text file corresponding to each of the data modes and upload the compressed text file to the target server; the target server is used to support data storage and data access.

[0032] As can be seen from the above technical solutions, the embodiments of this application have the following advantages:

[0033] This application provides a data conversion method, which includes: triggering the scheduling task when the triggering condition corresponding to the scheduling task is met, and transmitting the task parameters to the main program of the push task; querying target data in the database in a streaming mode based on the task parameters through the main program of the push task, parsing the target data according to the template file, and writing the target data into the local sharded data file; the template file here is generated in advance according to the structure of the data in the database; after querying the target data in the database and writing it into the local sharded data file, determining the total number of target data under each data mode queried; for each data mode, generating a text file corresponding to the data mode based on the total number of target data under the data mode queried and the local sharded data files under the data mode; compressing the text file corresponding to each data mode, and uploading the compressed text file to the target server, the target server being used to support data storage and data access. This method, by setting the triggering condition corresponding to the scheduling task, can effectively ensure the timeliness and accuracy of the read data, and by generating a template file by parsing the data structure in the database and parsing the data based on the template file, the coupling relationship between the database and the data consumer can be decoupled. Attached Figure Description

[0034] Figure 1 This is a schematic diagram of the structure of the data conversion system provided in the embodiments of this application;

[0035] Figure 2 A flowchart illustrating the data conversion method provided in this application embodiment;

[0036] Figure 3 A schematic diagram of the reverse process template list configuration file provided in the embodiments of this application;

[0037] Figure 4 Example illustration of the table template provided in this application embodiment;

[0038] Figure 5 Example diagram of decimal place template for table fields provided in the embodiments of this application;

[0039] Figure 6 A flowchart for generating template files provided in this application embodiment;

[0040] Figure 7 A flowchart illustrating the implementation of a scheduling task provided in an embodiment of this application;

[0041] Figure 8 This is a schematic diagram of the structure of the data conversion device provided in the embodiments of this application. Detailed Implementation

[0042] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present application.

[0043] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a particular order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0044] This application provides a data conversion system suitable for implementing the data conversion method provided in this application. The data conversion system will be described below. See also... Figure 1 , Figure 1This is a schematic diagram of the structure of the data conversion system provided in the embodiments of this application, such as... Figure 1 As shown, the system includes a template generation component, a task scheduling component, and a push program component.

[0045] The template generation component generates template files based on the data structure in the database and sends these target files to the data consumer. Specifically, it generates template files (such as Excel files) for each table based on the database's table definitions, field lengths, and field types, interpreting the data structure in text format. The generated template files are then synchronized with the data consumer to ensure data standardization.

[0046] The scheduling task component detects whether the triggering conditions for a scheduled task are met. Upon confirming that the triggering conditions are met, it triggers the scheduled task and transmits the task parameters to the main push task program. Specifically, it periodically scans the database and controls the start and stop of tasks based on data changes. Each scheduled task has its own corresponding triggering conditions, such as defined triggering parameters and triggering time. Scheduled tasks can be grouped and maintained, their execution status monitored, and recorded in corresponding logs.

[0047] The push notification component is one of the core components of this system. It is used by the main push task program to query target data in the database using a streaming mode based on the task parameters, parse the target data according to the template file, and write the target data into the landing shard data file. After querying the target data in the database and writing it into the landing shard data file, it determines the total number of target data for each data mode. For each data mode, it generates a text file corresponding to that data mode based on the total number of target data for that data mode and the landing shard data files for that data mode. It compresses the text file corresponding to each data mode and uploads the compressed text file to the target server. The target server supports data storage and data access.

[0048] Specifically, it is used to start through a scheduled task, query data in the database using Mybatis streaming mode, write the queried data into the local sharded data files according to the rules configured in the template file, merge and compress the local sharded data files, and upload them to an FTP (File Transfer Protocol) server for data consumers to access.

[0049] The data conversion method provided in this application will be described in detail below through method embodiments.

[0050] See Figure 2 , Figure 2 A flowchart illustrating the data conversion method provided in this application. Figure 2 As shown, this data transformation method includes the following steps:

[0051] Step 201: When the triggering conditions corresponding to the scheduling task are met, the scheduling task is triggered, and the task parameters are transmitted to the main program of the push task.

[0052] In practical applications, the scheduling center can set corresponding trigger conditions for each scheduling task, such as trigger time, trigger parameters, trigger node, etc. When the trigger conditions corresponding to the scheduling task are detected, the scheduling task can be triggered and the task parameters can be pushed to the main program of the push task.

[0053] In this embodiment, the scheduling initialization task can be flexibly configured to indicate either a full data push mode or an incremental data push mode. The full data push mode instructs the push of all current data in the database to the main push task program, while the incremental data push mode instructs the push of data that has been adjusted within a target time period to the main push task program. The target time period is the time between the current trigger time of the scheduling task and the previous trigger time. The above settings can be achieved simply by changing the push method and execution time in the scheduling initialization task.

[0054] It should be noted that each data record in the database is configured with a push identifier field. This push identifier field can indicate the push status of the corresponding data record, such as pending push, in the process of push, pushed by the last task, not pushed, pushed, etc. It is used to indicate which data in the database needs to be queried this time based on the push status of the data record in the database after the scheduling task of incremental data push mode is started.

[0055] It should be noted that the database also includes a deletion backup table corresponding to the data record table. This deletion backup table is used to record the data records that have been deleted from their respective data record tables. Since data in the database is usually deleted without a trace, this deletion backup table can be created to identify which data was deleted within a certain period of time during data transformation.

[0056] In addition, the scheduling task phase can also support functions such as executing shell scripts, clearing and deleting backup table data, and clearing server historical data files.

[0057] Step 202: Through the main program of the push task, based on the task parameters, query the target data in the database in streaming mode, parse the target data according to the template file, and write the target data into the landing shard data file; the template file is generated in advance according to the structure of the data in the database.

[0058] The scheduling task transmits the task parameters to the main program of the push task, which will trigger the main program of the push task to start. Then, the main program of the push task can query the target data in the database in streaming mode based on the task parameters (when the scheduling task is used to indicate the full data push mode, the target data is the current full data in the database; when the scheduling task is used to indicate the incremental data push mode, the target data is the data in the database that has changed relative to the last data transformation operation), and parse the target data according to the template file and write the target data into the local fragmented data file.

[0059] It should be noted that the aforementioned template file is pre-generated based on the structure of the data in the database. Specifically, the template file can be generated by reverse engineering the code from the template, based on the corresponding data tables of each airline in the database. This template file is used to interpret the data structure of the database in text form, and it is sent to the data consumer; that is, this template file serves as the standard for pushing data. Theoretically, one template file is generated for each data table when generating the data template. However, due to the special nature of domestic airfares, only one template file is generated for the same data table from multiple airlines, and this template file only provides business-related fields.

[0060] In one possible implementation, the main program for the push task can configure processing threads and database connections based on task parameters and hardware resources. The number of processing threads here is the maximum number of processing threads that the hardware resources can support, and the number of database connections here is the maximum number of database connections that the hardware resources can support. Then, through the configured database connections and processing threads, the target data in the database is queried concurrently in streaming mode, and the target data is parsed according to the template file and written to the local sharded data file.

[0061] Specifically, after the main push program is started by scheduling the task, the main push program will configure the maximum number of processing threads and the maximum number of database connections according to the input task parameters and hardware resources. Then, the query of multiple data tables will be split into multiple processing threads to be executed concurrently. By using technologies such as semaphores, the server hardware resources will be utilized to the maximum extent, thereby achieving the purpose of efficient push.

[0062] In addition, this application also adopts Mybatis streaming mode to query the database and write to the persistent sharded data file, and uses multiple processing threads to perform query and write operations simultaneously, thereby improving file writing efficiency and reducing server memory consumption.

[0063] Step 203: After querying the target data in the database and writing it into the landing shard data file, determine the total number of target data for each data mode queried; for each data mode, generate a text file corresponding to the data mode based on the total number of target data for the queried data mode and each landing shard data file for the data mode.

[0064] After all target data in the database has been queried and written to the local sharded data files, a shell script can be called to query the total number of target data (i.e., the total number of data records) for each data mode. Then, based on the total number of target data for each data mode and the local sharded data files for each data mode, a text file corresponding to each data mode can be generated.

[0065] Specifically, for each data pattern, a shell script can be invoked to merge the various data fragments within that data pattern into a single file corresponding to that data pattern, and then write the total number of target data for that data pattern to the top of the corresponding text file. The advantage of performing this operation via a shell script is that it bypasses JVM virtual machine resource limitations, performing the merging and compression directly at the operating system level, significantly improving execution efficiency.

[0066] Step 204: Compress the text file corresponding to each of the data patterns, and upload the compressed text file to the target server; the target server is used to support data storage and data access.

[0067] After obtaining the text file corresponding to each data pattern, the text file corresponding to each data pattern can be compressed into a zip file and uploaded to the target server (i.e., an FTP server, which can support data storage and data access) so that the data consumer can retrieve the data it needs from the FTP server.

[0068] In this embodiment, multiple scheduling tasks typically need to be executed concurrently. The text file generated by each scheduling task in response to the push task main program can be regarded as the candidate compressed file corresponding to that scheduling task. After uploading the subsequent compressed files corresponding to each scheduling task to the FTP server, the file format of each candidate compressed file can be adjusted to confirm the completion of this push task.

[0069] Specifically, after the main program of the push task generates a compressed text file, it needs to be uploaded to an FTP server for data consumers to use. This is because this solution typically involves multiple scheduled tasks running concurrently, and a push is considered successful only when all tasks complete successfully. Therefore, the compressed text files that complete the task first are uploaded to the FTP server with the .tmp extension. Once the last task is completed, the file extension of all compressed text files in this batch is changed back to .zip, indicating that the task is finally complete.

[0070] Furthermore, the solutions provided in this application embodiment extensively utilize Redis distributed caching to determine the scheduling task status and relationships between multiple processing threads.

[0071] The data conversion method provided in this application includes: triggering the scheduling task when the triggering condition corresponding to the scheduling task is met, and transmitting the task parameters to the push task main program; querying target data in the database using a streaming mode based on the task parameters through the push task main program, parsing the target data according to a template file, and writing the target data into a localized sharded data file; the template file is generated in advance based on the structure of the data in the database; after querying the target data in the database and writing it into the localized sharded data file, determining the total number of target data under each data mode; for each data mode, generating a text file corresponding to that data mode based on the total number of target data under that data mode and the localized sharded data files under that data mode; compressing the text file corresponding to each data mode, and uploading the compressed text file to the target server, the target server being used to support data storage and data access. This method, by setting the triggering condition corresponding to the scheduling task, can effectively ensure the timeliness and accuracy of the read data. Furthermore, by generating a template file by parsing the data structure in the database and parsing the data based on the template file, the coupling relationship between the database and the data consumer can be decoupled.

[0072] To facilitate a further understanding of the data conversion method provided in the embodiments of this application, the data conversion method provided in the embodiments of this application will be described in its entirety as an example below.

[0073] First, we will introduce the modification and application of data structures in databases.

[0074] Specifically, in this embodiment of the application, the following two fields are added to the business table stored in the database to mark the push status of data records:

[0075]

[0076] A new field, PUSH_FLAG, is added. During update operations, the Mybatis interceptor in DFF, along with a custom annotation, is used to determine whether the update operation involves a table with a newly added field. The update method for tables with newly added fields is intercepted, and the PUSH_FLAG field is reassigned by modifying the update method.

[0077] The newly added field FIRST_PUSHED is assigned a default value of 0 during the new operation. After the push program successfully pushes this data record, the value is changed to 1 to mark this record as having been pushed.

[0078] Create a deletion backup table corresponding to the business table in each database schema. The deletion backup table retains the original business fields and adds two new fields at the beginning.

[0079] Field Name Field Description Field type Allows empty Remark D_UUID Primary key VARCHAR2(32) N Default value: UUID DELETE_TIME Record deletion time DATE N Default value: sysdate

[0080] Naming conventions for deleting backup tables: Use "D_" as a prefix, followed by the original business table name, and take the first 30 characters as the table name.

[0081] For example, the original table name is RECORDS8_DETAIL, and after conversion it becomes D_RECORDS8_DETAIL.

[0082] When performing a physical delete operation on a business data table, the Mybatis interceptor in DFF, along with custom annotations, determines whether the delete statement needs to be split. If so, it will split into two statements: one inserts the record to be deleted into the backup table, and the second is the original physical delete statement, ensuring that both statements are managed and controlled within the same transaction.

[0083] The PUSH_FLAG field in the backup deletion table is defined as follows, and the status value of this field at each stage is maintained by the main program of the push task.

[0084]

[0085] Next, we will provide an example of how to generate template files.

[0086] 1. Compile the table names and table IDs that need to be pushed into an Excel file.

[0087] 2. Read Excel data into a collection using a program.

[0088] 3. Loop through the collection.

[0089] 4. Once the schema name and table name are obtained, use SQL statements to read the table structure information (e.g., field names, field types, field lengths, etc.).

[0090] 5. Write data into the template. Add a char attribute m_prefixId with a length of 3 at the beginning of each table for unique identification, and add a char attribute m_actionCode with a length of 1 to identify the meaning of the data. Add a field m_cxrCode with a length of 7 to all tables under the airline pattern to identify the airline's two-letter code.

[0091] 6. Traverse the table structure and write the attribute name, starting field length, ending length, field type, value range, and field description into Excel in sequence. Attribute names should follow camelCase, with the first letter lowercase and prefixed with "m_". Attribute types and lengths should be calculated by adding the actual length of the field in the database. String types are incremented by 1, numeric types by 2, and date types have a fixed length of 14. (Special handling: In Oracle, these are number(1), char(1), and varchar(1), all corresponding to the char attribute type with a length of 1).

[0092] 7. Set HEAD, and set the class name (ClassName) to camelCase (schema name + table name), with the first letter capitalized, for example, FCIMS.DFIM_T_CAT04 --> FcimsDfimTCat04; set the attribute name (tableName) to all uppercase (schema name + table name), for example, FCIMS.DFIM_T_CAT04; set the xmlName to lowercase (table name), for example, auto_cabin_class; set rowLength to the total length of all fields in the table.

[0093] 8. Generate an Excel template file, named in camelCase format (schema name + table name), with the first letter capitalized.

[0094] 9. Simultaneously, during the generation of the table template file, record all small number segment information and store the precision and scale of the small number segments in a Map.

[0095] 10. After all table templates are generated, generate table field decimal place templates based on the previously recorded decimal segment information.

[0096] Reverse process template list configuration file such as Figure 3 As shown. The RecordName column contains the data schema name + table name.xls file. Pascal's notation is used, removing underscores from the names; the schema name is treated as a single word. The tableId column is a predefined number for each table.

[0097] Example of a table template: Figure 4As shown. The template file name convention is data schema name + table name.xls. Pascal's notation is used, removing underscores from the name, and the schema name is treated as a single word. For example, the NFS_TA.FBR_DTL_ROUTE table is converted to NfstaFbrDtlRoute.xls.

[0098] The first row is the table header row.

[0099] A1: Same filename, but without the extension.

[0100] B1: Total length of the current form records.

[0101] C1: Data schema name.table name. Same as the naming convention in the database. For example, NFS_TA.FBR_DTL_ROUTE.

[0102] D1: Table name in all lowercase. Example: fbr_dtl_route after NFS_TA.FBR_DTL_ROUTE conversion.

[0103] A3 and below: Parameter naming definition. The parameter prefix is ​​"m_" followed by the business field name of the data table (camel case).

[0104] The first three fixed parameters are explained as follows:

[0105] m_prefixId: The number defined for the current table, which can be found in the table's primary key template document.

[0106] m_actionCode: The currently recorded action.

[0107] m_cxrCode: Only needs to be defined in NFS_XX mode. It is a two-letter code for the airline, such as CA, CZ, etc.

[0108] Below B3: Starting position of field value, required field.

[0109] C3 and below: Field value cutoff position, required field.

[0110] D3 and below: Field type. Only three data types are allowed: char, datetime, and int. Required field.

[0111] E3 and below: Sample values. Only E4 is required; the others are optional.

[0112] F3 and below: Field descriptions. Not required.

[0113] Table field decimal place template example Figure 5 As shown. The template file name is: percent.xls;

[0114] The first row is the table header row.

[0115] Column A: Table name is the same as the data template file name.

[0116] Column B: Field names.

[0117] Column C: Same as Column B.

[0118] Column D: Total field length.

[0119] Column G: Integer length.

[0120] Column H: Decimal place length.

[0121] Column J: Default -9999.

[0122] The flowchart for generating the template file is as follows: Figure 6 As shown.

[0123] Next, a detailed example of the implementation of the scheduling task will be provided. Figure 7 This is a flowchart illustrating the process of implementing this scheduling task.

[0124] 1. In the timed scheduling center, maintain configuration information such as task type and trigger time for different timed scheduling tasks.

[0125] 2. After a scheduled task is triggered, the main program for pushing tasks is called. Then, it checks if there are any scheduled tasks currently being executed. If so, the execution of the current scheduled task is canceled, and the task waits for the next scheduled task to be executed.

[0126] 3. Set the batch number of the scheduled task to be executed this time, in the format yyyyMMddHHmmss (accurate to the minute).

[0127] 4. Check whether a valid batch number has been entered for the scheduling task.

[0128] 5. Check the task type of this push (C - incremental, A - full).

[0129] 6. Set different Redis cache data according to different task types that are triggered.

[0130] 7. Set the batch number for this push task in Redis. The key is application name:push task name:push task type:batch number suffix (e.g., DFP:DPRIC:A:BATCH:NUMBER).

[0131] 8. Set the push task type in Redis, with the key being application name:push task name:TASKNAME (e.g., DFP:TASKNAME).

[0132] 9. Set Redis to incremental / full type, with the key being application name:push task name:TRANS (e.g., DFP:TRANS).

[0133] 10. Set the total number of each mode in Redis, with the key being application name:push task name:push data type:batch number suffix (e.g., DFP:DPRIC:A:20210709100000).

[0134] 11. The push task begins execution.

[0135] 12. Obtain the push task type, push type, and batch number.

[0136] 13. Check again whether it is from the same batch. If it is not from the same batch, throw a "A program is detected to be running" exception.

[0137] 14. Read the output parameters to obtain the current mode, table name array, and airline array.

[0138] 15. Set the total number of tables for each Redis mode, with the key being: application module: push task name: push data type: batch number: database mode name (e.g., DFP:DPRIC:A:20210709100000:FCIMS).

[0139] 16. Use the CountDownLatch synchronous controller to record the total number of tables for each pattern.

[0140] 17. Loop through table data. First, determine if it is an airline mode. If it is an airline mode, use a nested loop to loop through the airlines first, and then read the table data.

[0141] 18. Construct a data information object, including the push task name, push data type, data mode name, airline code, mode + table name, table name, stream, set the maximum number of rows to read in the stream mode, and push data batch number.

[0142] 19. If the push mode is the airline mode, a Semaphore needs to be introduced to maintain the maximum number of currently executable threads. Using a Semaphore can control the number of threads accessing resources at the same time, maximize the utilization of CPU performance, help the correctness of program logic, avoid non-thread safety issues, and ensure the stable operation of system functions.

[0143] 20. Start the data processing method.

[0144] 21. If handling the airline pattern, save the currently executing airline code to the context, and wait for the Mybatis interceptor to replace the airline code.

[0145] 22. Modify the actual table name being processed using Mybatis interceptors. For example, if the table being processed is for MU Airlines, the schema name NFS_CA in the SQL statement will be converted to NFS_MU.

[0146] 23. Obtain the ClassBeanName of the full path mapping Mapper and Example class (e.g., com.travelsky.dff.message.dao.nfs_ta.BaseFareMapper) through the converted table name.

[0147] 24. Obtain a flow class instance object and execute the file writing operation asynchronously.

[0148] 25. Use reflection to obtain the Mapper through the converted ClassBeanName.

[0149] 26. Obtain the updateByPushFlag method from the business table DAO by reflection, change the push_flag field from 0 to 1, and mark the current data as being processed.

[0150] 27. Obtain the template object of the current table by task type, schema name, and table name, which is used to generate sharded data files.

[0151] 28. Change the status of deleting backup table data to "processing" and change the value of push_flag (which is less than 3) to 3.

[0152] 29. Obtain an instance of the flow processing class to handle operations related to deleting backup tables.

[0153] 30. Set the actionCode (record push type) in the data information object to (X) and delete it.

[0154] 31. Place the data information object, data template, and number of data rows for each shard into the flow object.

[0155] 32. Define a Phaser, which stands for "phaser," to solve the problem of controlling multiple threads to complete a task in stages.

[0156] 33. Get the table name of the deleted backup table.

[0157] 34. Obtain the ClassBeanName of the full path mapping Mapper and Example class (e.g., com.travelsky.dff.message.dao.nfs_ta.BaseFareMapper) through the converted table name.

[0158] 35. Obtain the updateByPushFlag method through reflection and modify the push_flag field from <3 to 3.

[0159] 36. Obtain the FlowSelectForDpric method through reflection. If it is NFS_XX, the airline parameter needs to be added.

[0160] 37. Pass in the query start time, end time, airline, and FlowResultHandler to execute the method to retrieve the deleted backup table data.

[0161] 38. If the number of data records obtained is greater than 0, write the schema name and table name to Redis.

[0162] 39. Change the actionCode field to null.

[0163] 40. Define the Phaser.

[0164] 41. Use reflection to call the flowSelectForDpric method to obtain the business table data for this task.

[0165] 42. If the number of data rows in the business table is greater than 0, then write the schema name + table name to Redis.

[0166] 43. Execute concurrent processing of data returned from the stream mode and write it to the sharded file.

[0167] 44. Save the airline code for the current thread.

[0168] 45. Start the method to write fragmented data.

[0169] 46. ​​Loop through the data object, parse the template object, and transform the data content according to the template rules.

[0170] 47. Define a StringBuffer string to concatenate the contents of a record.

[0171] 48. Concatenate fixed parameter text content. If the template content is not a fixed parameter, terminate the loop. If it is a calculated field length, concatenate it into the string and pad with spaces on the right according to the field length.

[0172] 49. Get the actionCode of the current record. The default is a new record. Convert the object field to a Map. If it is an incremental push, the data record type of the record needs to be calculated. If the actionCode value in the data object is X, it means that the current record being processed is a deletion record. Get the first_pushed field. If the first_pushed field value is 0, it is a new record and the actionCode value is I. Otherwise, it is an update and the value is set to C.

[0173] 50. Concatenate the business parameter text content, convert the object value to a MAP, iterate through the template parameters and perform value writing and string conversion operations. If it is a fixed parameter, continue the loop. Remove the 'm_' prefix from the template parameters and convert them to camelCase. Determine if it is a valid parameter (including business parameters and fixed parameters), calculate the parameter length, and determine the parameter type. If it is a numeric type, convert it to a string; if it is a date type, convert it to a string in the format "yyyyMMddHHmmss". Then, pad the right side with half-width spaces according to the field length.

[0174] 51. Concatenate the business data text file paths.

[0175] 52. Continue writing the file content after completing the data in other tables.

[0176] 53. The Phaser waits for other participants to complete.

[0177] 54. Check if the data files for all tables in the current mode have been generated. If so, merge, compress, and push the data.

[0178] 55. By querying Redis data, check if the total number of tables processed matches the expected total number of tables for keys ending in TABLE:LIST or TABLE:TOTAL. If they match, it means that all table shard data files have been generated in the current mode.

[0179] 56. Obtain the server address of the generated data file, and obtain the total number of data records generated in the current mode of this batch by executing a shell script.

[0180] 57. The total record count is padded with zeros on the left, and the fixed length is 9. Create a Header object and write the current mode name, transmission type, transmission time, and total record count information into the header file.

[0181] 58. Generate the push data file for the current mode.

[0182] 59. Concatenate the header information file path and write the Header object to the header information file.

[0183] 60. After the push data file is generated, obtain the file's path and name, call the Shell script to merge the split files into a txt data file, and then compress the txt data file into a zip file.

[0184] 61. Upload the generated zip data file with the .tmp extension to the specified directory on the specified FTP server.

[0185] 62. Call the Shell script to clean up all fragmented data files and merged txt data files in the current batch and current mode.

[0186] 63. By comparing the Redis key, SCHEMA_LIST, and SCHEMA_TOTAL, determine whether the total number of completed patterns matches the expected total number of patterns. If they match, it indicates that all data files for this batch have been generated.

[0187] 64. Modify the push_flag and first_pushed fields. Change the push_flag field of the business table from 1 to 2, and the first_pushed field from 0 to 1. Change the push_flag field of the deleted backup table from 3 to 4.

[0188] 65. Delete all cached data in Redis for this batch.

[0189] 66. Change all .tmp files on the FTP server to .zip compressed files.

[0190] 67. This batch of push notifications is complete.

[0191] The technical solution provided in this application, by adding a push identifier field and establishing a deletion backup table to the original data structure, and in conjunction with custom scanning rules, achieves the purpose of capturing and processing data change records (incremental data) within a certain period. Furthermore, it utilizes AOP slicing technology to split specified SQL statements in the DAO layer operations, and through their respective rule-based splitting methods, completes the maintenance of push identifiers and the registration of deletion records. This solution involves zero intrusion into business code and greatly simplifies the work of developers, allowing them to focus entirely on the implementation of business logic.

[0192] Furthermore, by employing Mbatis streaming mode for data querying and utilizing technologies such as multithreading, synchronization controllers, stagers, and semaphores, a single server can perform multithreaded concurrent read and write operations on multiple modes and multiple files. This significantly improves data processing efficiency, with an average CPU utilization exceeding 95% during task execution. The component also allows for configuration of the number of threads and the batch size of data processing, enabling flexible allocation based on different hardware resource conditions.

[0193] Furthermore, using Excel template files to define the data file structure decouples the code from the data parsing rules, resulting in high readability, a large amount of configuration information, and ease of maintenance. This solution also provides the ability to generate template files through reverse engineering, significantly reducing the enormous workload and the possibility of errors associated with manual coding. Future changes to the data file content require only modifications to the template content, without altering the code. This reduces maintenance costs and greatly improves response speed.

[0194] This application also provides a data conversion apparatus. See [link to previous document]. Figure 8 , Figure 8 This is a schematic diagram of the structure of the data conversion device provided in the embodiments of this application, such as... Figure 8 As shown, the device includes:

[0195] The scheduling task triggering module 801 is used to trigger the scheduling task when the triggering conditions corresponding to the scheduling task are met, and transmit the task parameters to the push task main program.

[0196] The data parsing module 802 is used to query the target data in the database in streaming mode based on the task parameters through the main program of the push task, parse the target data according to the template file, and write the target data into the landing segmented data file; the template file is generated in advance according to the structure of the data in the database.

[0197] The file generation module 803 is used to query the target data in the database and write it into the landing shard data file, and then determine the total number of target data for each data mode queried; for each data mode, it generates a text file corresponding to the data mode based on the total number of target data for the data mode queried and the landing shard data files for each data mode.

[0198] The file upload module 804 is used to compress the text file corresponding to each of the data modes and upload the compressed text file to the target server; the target server is used to support data storage and data access.

[0199] Optionally, the data parsing module 802 is specifically used for:

[0200] The main program for the push task configures processing threads and database connections based on the task parameters and hardware resources. The number of processing threads is the maximum number of processing threads that the hardware resources can support, and the number of database connections is the maximum number of database connections that the hardware resources can support.

[0201] Through the database connection and the processing thread, the target data in the database is queried concurrently in streaming mode, and the target data is parsed according to the template file and written into the local sharded data file.

[0202] Optionally, the triggering conditions corresponding to the scheduling task include at least one of triggering time, triggering parameters, and triggering node;

[0203] The scheduling task is used to indicate either a full data push mode or an incremental data push mode; the full data push mode is used to indicate that the current data in the database is pushed to the main program of the push task, and the incremental data push mode is used to indicate that the data in the database that has been adjusted within a target time period is pushed to the main program of the push task, and the target time period is the time period between the time when the scheduling task is triggered this time and the time when the scheduling task was triggered last time.

[0204] Optionally, when multiple scheduling tasks are executed concurrently, the compressed text file generated by the main program of the push task for each scheduling task is used as a candidate compressed file; the file upload module 804 is further configured to:

[0205] After the main program of the push task generates candidate compressed files corresponding to each of the scheduling tasks and uploads the candidate compressed files corresponding to each of the scheduling tasks to the target server, the file format of each candidate compressed file is adjusted, and the push task is confirmed to be completed.

[0206] Optionally, the apparatus further includes a template file generation module; the template file generation module is used for:

[0207] The template file is generated by reverse engineering the code using the template and based on the data tables corresponding to each airline in the database. The template file is used to interpret the data structure in the database in text form and will be sent to the data consumer.

[0208] Optionally, the file generation module 803 is specifically used for:

[0209] For each of the data patterns, a shell script is invoked to merge the various data fragments in the data pattern into a file corresponding to the data pattern, and the total number of target data in the data pattern is written to the top of the text file.

[0210] Optionally, each data record in the database is configured with a push identifier field; the push identifier field is used to indicate the push status corresponding to the data record.

[0211] Optionally, the database also includes a deletion backup table corresponding to the data record table; the deletion backup table is used to record the data records deleted from its corresponding data record table.

[0212] The aforementioned device can effectively ensure the timeliness and accuracy of the read data by setting the trigger conditions corresponding to the scheduling task. Furthermore, by generating a template file by parsing the data structure in the database and parsing the data based on the template file, the coupling relationship between the database and the data consumer can be decoupled.

[0213] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0214] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection between apparatuses or units through some interfaces, and may be electrical, mechanical, or other forms.

[0215] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0216] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0217] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes: USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, optical disks, and other media capable of storing computer programs.

[0218] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0219] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A data conversion method, characterized in that, The method includes: When the triggering conditions corresponding to the scheduling task are met, the scheduling task is triggered, and the task parameters are transmitted to the main program of the push task. The main program for the push task queries the target data in the database in streaming mode based on the task parameters, parses the target data according to the template file, and writes the target data into the local segmented data file; the template file is generated in advance according to the structure of the data in the database. After querying the target data in the database and writing it into the landing shard data file, the total number of target data for each data mode queried is determined; for each data mode, a text file corresponding to the data mode is generated based on the total number of target data for the data mode queried and the landing shard data files for each data mode. Compress the text file corresponding to each of the data patterns, and upload the compressed text file to the target server; the target server is used to support data storage and data access. The template file was generated in the following way: The template file is generated by reverse engineering the code using the template and based on the data tables corresponding to each airline in the database. The template file is used to interpret the data structure in the database in text form and will be sent to the data consumer.

2. The method according to claim 1, characterized in that, The process of querying target data in the database using a streaming mode based on the task parameters through the main program of the push task, parsing the target data according to the template file, and writing the target data into the landing segmented data file includes: The main program for the push task configures processing threads and database connections based on the task parameters and hardware resources. The number of processing threads is the maximum number of processing threads that the hardware resources can support, and the number of database connections is the maximum number of database connections that the hardware resources can support. Through the database connection and the processing thread, the target data in the database is queried concurrently in streaming mode, and the target data is parsed according to the template file and written into the local sharded data file.

3. The method according to claim 1, characterized in that, The triggering conditions corresponding to the scheduling task include at least one of triggering time, triggering parameters, and triggering node; The scheduling task is used to indicate either a full data push mode or an incremental data push mode; the full data push mode is used to indicate that the current data in the database is pushed to the main program of the push task, and the incremental data push mode is used to indicate that the data in the database that has been adjusted within a target time period is pushed to the main program of the push task, and the target time period is the time period between the time when the scheduling task is triggered this time and the time when the scheduling task was triggered last time.

4. The method according to claim 1, characterized in that, When multiple scheduling tasks are executed concurrently, the compressed text file generated by the main program of the push task for each scheduling task is used as a candidate compressed file. The method further includes: After the main program of the push task generates candidate compressed files corresponding to each of the scheduling tasks and uploads the candidate compressed files corresponding to each of the scheduling tasks to the target server, the file format of each candidate compressed file is adjusted, and the push task is confirmed to be completed.

5. The method according to claim 1, characterized in that, For each data pattern, based on the total number of target data under the queried data pattern and the data shards of each data pattern, a text file corresponding to the data pattern is generated, including: For each of the data patterns, a shell script is invoked to merge the various data fragments in the data pattern into a text file corresponding to the data pattern, and the total number of target data in the data pattern is written to the top of the text file.

6. The method according to claim 1, characterized in that, Each data record in the database is configured with a push identifier field; the push identifier field is used to indicate the push status corresponding to the data record.

7. The method according to claim 1, characterized in that, The database also includes a deletion backup table corresponding to the data record table; the deletion backup table is used to record the data records that have been deleted from the corresponding data record table.

8. A data conversion system, characterized in that, The system includes: A template generation component is used to generate template files based on the structure of data in the database and send the template files to the data consumer. The scheduling task component is used to detect whether the triggering conditions corresponding to the scheduling task are met, and when it is determined that the triggering conditions corresponding to the scheduling task are met, the scheduling task is triggered and the task parameters are transmitted to the push task main program. The push notification component is used to query target data in the database using a streaming mode based on the task parameters, according to the main push task program; parse the target data according to the template file; write the target data into the landing shard data file; after querying the target data in the database and writing it into the landing shard data file, determine the total number of target data for each data mode; for each data mode, generate a text file corresponding to the data mode based on the total number of target data for the data mode and the landing shard data files for the data mode; compress the text file corresponding to each data mode and upload the compressed text file to the target server; the target server is used to support data storage and data access. The template generation component is specifically used to reverse engineer the code using a template to generate the template file based on the data tables corresponding to each airline in the database; the template file is used to interpret the data structure in the database in text form.

9. A data conversion device, characterized in that, The device includes: The scheduling task triggering module is used to trigger the scheduling task when the triggering conditions corresponding to the scheduling task are met, and transmit the task parameters to the push task main program; The data parsing module is used to query the target data in the database in streaming mode based on the task parameters through the main program of the push task, parse the target data according to the template file, and write the target data into the landing segmented data file; the template file is generated in advance according to the structure of the data in the database. The file generation module is used to query the target data in the database and write it into the landing shard data file, and then determine the total number of target data for each data mode queried; for each data mode, based on the total number of target data for the data mode queried and the landing shard data files for each data mode, a text file corresponding to the data mode is generated. The file upload module is used to compress the text file corresponding to each of the data modes and upload the compressed text file to the target server; the target server is used to support data storage and data access. The device further includes a template file generation module; the template file generation module is used for: The template file is generated by reverse engineering the code using the template and based on the data tables corresponding to each airline in the database. The template file is used to interpret the data structure in the database in text form and will be sent to the data consumer.

Images