A file generation method, device and storage medium

CN112925743BActive Publication Date: 2024-11-05INDUSTRIAL AND COMMERCIAL BANK OF CHINA
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202110168147.1
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-02-07
Publication Date
2024-11-05
Estimated Expiration
2041-02-07

AI Technical Summary

Technical Problem

这种模式已经无法满足业务对于系统处理时效以及业务功能上线周期的需求

Benefits of technology

[0011] It can be seen from the technical solutions provided by the above embodiments of this specification that in the embodiments of this specification, a file generation task can be obtained; the file generation task includes converting the target data in the first data table into a file that conforms to the structure of the second data table; The file generation task corresponds to a file configuration table; the file configuration table at least includes the change date of the second data table structure and the current column information of the second data table; after obtaining the current date of the file generation task If the date of change is earlier than the change date of the second data table structure, the target data is converted according to the current column information of the second data table to generate a file that conforms to the second data table structure. The method provided by the embodiments of this specification can achieve vertical expansion of the system through the file configuration table, making it easier to expand and maintain large-scale file generation tasks, and ensuring that when the upstream table structure changes, a file format that adapts to the downstream table structure is provided in a timely manner. , thereby improving the efficiency of file generation.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN112925743B_ABST
    Figure CN112925743B_ABST
Patent Text Reader

Abstract

The embodiments of this specification provide a file generation method, device and storage medium, which can be applied to the field of big data processing technology. The method includes: obtaining a file generation task; the file generation task includes converting target data in a first data table into a file that conforms to a second data table structure; the file generation task corresponds to a file configuration table; the file configuration table includes at least a change date of the second data table structure and current column information of the second data table; when the current date of obtaining the file generation task is earlier than the change date of the second data table structure, converting the target data into a file that conforms to the second data table structure according to the current column information of the second data table to improve the efficiency of file generation.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The embodiments of the present specification relate to the field of big data processing technology, and in particular to a file generation method, device and storage medium. Background Art

[0002] When migrating data from traditional relational databases to the Hadoop platform, data migration is still mainly done in batch file exchange. This is because the upstream file structure must be strictly consistent with the table structure of the downstream big data database (including hive and hbase databases), and after the upstream table structure changes, the downstream table structure must be changed immediately to achieve consistency between the upstream and downstream table structures. Therefore, a file generation system that can support configuration and easy expansion is required, and the file system must also be able to identify changes in the upstream source table structure.

[0003] Currently, the most commonly used file generation methods include serial generation of fixed structure files and configuration generation based on the traditional relational database Oracle database view USER_TAB_COLUMNS or the MySQL database information_schema.columns.

[0004] The method of generating fixed structure files serially is to generate files with fixed structures to make the upstream and downstream file structures consistent. However, this method is not flexible to expand, has strong dependencies between files, takes a long time to execute in batches, and cannot identify changes in the source table structure. The configuration generation method based on the traditional relational database Oracle database view USER_TAB_COLUMNS or the Mysql database information_schema.columns reflects the changes in the source table structure in the file structure immediately, but cannot support the scenario where there is a time lag in the synchronization of downstream table structure changes.

[0005] At present, in response to the scenario where the upstream source table structure changes affect the downstream data loading, the emergency solution of changing the data date and temporarily downloading empty files is usually adopted. This mode can no longer meet the business requirements for system processing timeliness and business function launch cycle. In order to improve business efficiency and control level, it is urgent to design a flexible and efficient file generation system. Summary of the invention

[0006] The purpose of the embodiments of this specification is to provide a file generation method, device and storage medium to improve the efficiency of file generation.

[0007] To solve the above problem, an embodiment of the present specification provides a file generation method, the method comprising: obtaining a file generation task; the file generation task comprises converting target data in a first data table into a file conforming to a second data table structure; the file generation task corresponds to a file configuration table; the file configuration table comprises at least a change date of the second data table structure and current column information of the second data table; in a case where the current date of obtaining the file generation task is earlier than the change date of the second data table structure, converting the target data into a file conforming to the second data table structure according to the current column information of the second data table.

[0008] To solve the above problems, an embodiment of the present specification also provides a file generation device, which includes: an acquisition module, used to acquire a file generation task; the file generation task includes converting the target data in the first data table into a file that conforms to the second data table structure; the file generation task corresponds to a file configuration table; the file configuration table at least includes the change date of the second data table structure and the current column information of the second data table; a generation module, used to convert the target data into a file that conforms to the second data table structure according to the current column information of the second data table when the current date of acquiring the file generation task is earlier than the change date of the second data table structure.

[0009] To solve the above problems, an embodiment of the present specification also provides an electronic device, including: a memory for storing a computer program; a processor for executing the computer program to implement: obtaining a file generation task; the file generation task includes converting target data in a first data table into a file that conforms to a second data table structure; the file generation task corresponds to a file configuration table; the file configuration table includes at least a change date of the second data table structure and current column information of the second data table; in a case where the current date of obtaining the file generation task is earlier than the change date of the second data table structure, converting the target data into a file that conforms to the second data table structure according to the current column information of the second data table.

[0010] To solve the above problems, an embodiment of the present specification also provides a computer-readable storage medium on which computer instructions are stored, and when the instructions are executed, the following are achieved: obtaining a file generation task; the file generation task includes converting target data in a first data table into a file that conforms to a second data table structure; the file generation task corresponds to a file configuration table; the file configuration table includes at least a change date of the second data table structure and current column information of the second data table; in a case where the current date of obtaining the file generation task is earlier than the change date of the second data table structure, converting the target data into a file that conforms to the second data table structure according to the current column information of the second data table.

[0011] It can be seen from the technical solutions provided in the above embodiments of this specification that, in the embodiments of this specification, a file generation task can be obtained; the file generation task includes converting the target data in the first data table into a file that conforms to the structure of the second data table; the file generation task corresponds to a file configuration table; the file configuration table includes at least the change date of the second data table structure and the current column information of the second data table; in the case where the current date of the file generation task is earlier than the change date of the second data table structure, the target data is converted to generate a file that conforms to the second data table structure according to the current column information of the second data table. The method provided in the embodiments of this specification can realize the vertical expansion of the system through the file configuration table, making the expansion and maintenance of large batch file generation tasks easy to achieve, ensuring that when the upstream table structure changes, the file format that adapts to the downstream table structure is provided in time, thereby improving the efficiency of file generation. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] In order to more clearly illustrate the embodiments of this specification or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this specification. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0013] Figure 1 This is a structural diagram of the upstream and downstream data system of the embodiment of this specification;

[0014] Figure 2 This is a schematic diagram of the structure of a file generation system in a scenario example of this specification;

[0015] Figure 3 This is a program flow chart of a task scheduling module in a scenario example of this specification;

[0016] Figure 4 This is a program flow chart of a concurrent job control module in a scenario example of this specification;

[0017] Figure 5 This is a program flow chart of a file generation module in a scenario example of this manual;

[0018] Figure 6 This is a program flow chart of a table structure change identification module in a scenario example of this specification;

[0019] Figure 7 This is a flowchart of a method for generating a file according to an embodiment of the present specification;

[0020] Figure 8This is a functional structure diagram of an electronic device according to an embodiment of this specification;

[0021] Fig. 9 This is a functional structure diagram of a file generating device according to an embodiment of this specification. DETAILED DESCRIPTION

[0022] The following will be combined with the drawings in the embodiments of this specification to clearly and completely describe the technical solutions in the embodiments of this specification. Obviously, the described embodiments are only part of the embodiments of this specification, not all of the embodiments. Based on the embodiments in this specification, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of this specification.

[0023] At present, in response to the scenario where the upstream source table structure changes affect the downstream data loading, the emergency plan of changing the data date and temporarily downloading empty files is usually adopted. This mode can no longer meet the business requirements for system processing timeliness and business function launch cycle. Considering that if a table is pre-configured and the information of the upstream data table and downstream data table involved in the file generation task is configured in this table, when performing the file generation task, it can be determined whether the upstream data table has changed based on the information in the table. This is expected to solve the problem that the upstream and downstream systems cannot meet the business requirements for system processing timeliness and business function launch cycle when migrating data through batch file exchange, and ensure that when the upstream table structure changes, the file format that adapts to the downstream table structure is provided in time, thereby improving the efficiency of file generation.

[0024] Figure 1 It is a structural diagram of the upstream and downstream data system. The file generation task of the embodiment of this specification is mainly the work process of migrating the data of the traditional relational database to the Hadoop platform. Data migration is mainly in the form of batch file exchange. Because the upstream file structure must be strictly consistent with the table structure of the downstream big data database (including hive and hbase databases), and after the upstream table structure changes, the downstream table structure must be changed immediately afterwards to achieve consistency between the upstream and downstream table structures.

[0025] This manual provides an example scenario of a file generation system. Figure 2 The file generation system 1 may include a file list configuration module 11, a task scheduling module 12, a concurrent job control module 13, a file generation module 14 and a table structure change identification module 15.

[0026] In some embodiments, the file list configuration module 11 may be used to configure a file configuration table. The file configuration table may include information of an upstream data table and a downstream data table involved in a file generation task, as shown in Table 1.

[0027] Table 1

[0028]

[0029] Among them, the target application system is the downstream system, and the data in the target application system can be stored through the downstream table; the source system is the upstream system, and the data in the source system can be stored through the upstream table.

[0030] In some embodiments, the task scheduling module 12 can be used to split and schedule file generation tasks. Figure 3 As shown, the task scheduling module 12 can perform the following steps.

[0031] S301: Initiate a file generation task.

[0032] In some embodiments, the file generation task may be to convert the target data in the upstream table into a file that conforms to the downstream table structure. The file generation task may be triggered manually by a user or in batches through the system background. After the system detects the file generation task, it may initiate a file generation task processing flow.

[0033] S302: creating subtasks cyclically according to batch dates.

[0034] In some embodiments, for a file generation task, the information in the file configuration table can be combined to construct subtasks according to the dimension of "table + batch date + batch", and parallel processing can be performed according to the granularity of the subtasks. Specifically, a file generation task may contain more content, such as converting the data in the upstream table A into a file that conforms to the structure of the downstream table A1, converting the data in the upstream table B into a file that conforms to the structure of the downstream table B1, etc., or the conversion date is different, etc. According to the information in the file configuration table, the data conversion of each row of the upstream table can be used as a subtask to complete the construction of the subtask.

[0035] S303: Determine from the task queue whether the subtask has been created.

[0036] Specifically, after executing S302, it can be determined whether there are subtasks to be generated. If there are subtasks to be generated, it is determined whether the task queue includes the subtask. If the subtask is included, the process is terminated; if the subtask is not included, the parameters are read from the file configuration table to create the subtask; if there are no subtasks to be generated, the process is terminated.

[0037] In some embodiments, the concurrent job control module 13 is used to control the concurrency of task execution. The concurrent processing of subtasks is realized through the job control module, which improves the throughput processing capacity of the system. Figure 4As shown, the concurrent job control module 13 can perform the following steps.

[0038] S401: Whether there are unstarted subtasks in the task queue.

[0039] If so, execute S402; otherwise, execute S404.

[0040] S402: Is the number of currently running subtasks less than 4?

[0041] In some embodiments, the concurrent job control module 13 may pre-set the concurrency, that is, the number of subtasks to be processed concurrently, for example, to 4, or other values, such as 3, 5, 6, etc.

[0042] If yes, execute S403; otherwise, execute S404.

[0043] S403: Start the subtask.

[0044] S404: Whether the subtasks of the current batch have been completed.

[0045] If yes, then initialize the next batch of subtasks; otherwise, continue to execute S401.

[0046] In some embodiments, the file generation module 14 is used to perform a file generation task. Figure 5 As shown, the file generation module 14 can perform the following steps.

[0047] S501: Read the file configuration table.

[0048] In some embodiments, before executing the file generation task, the file generation module 14 can read the information in the file configuration table, such as the current column information of the downstream table, whether the upstream table structure has changed, the date of the downstream table structure change, and the column information after the downstream table structure changes.

[0049] Before the file generation module 14 reads the file configuration table, the table structure change identification module 15 has identified the table structure changes one by one in the file configuration table and updated the "ISCHANGE" field. If it is identified that the same field has changed and needs special processing, the field is automatically updated to the "SPECIAL_COLS" field, and the processing logic of the corresponding column is updated to the "SPECIAL_COL_SQLS" field.

[0050] Specifically, after the upstream table structure changes, the downstream table structure is still used to provide files. Therefore, the table structure change identification module 15 identifies the table structure change, which can be compatible with the problem of delayed downstream table change time, and notify the downstream that the upstream table has changed and please change the downstream table structure as soon as possible. Figure 6 As shown, the table structure change identification module 15 can specifically perform the following steps.

[0051] S601: Initiate a table structure change identification task.

[0052] In some embodiments, since the file generation task is usually scheduled in the early morning when the system is relatively idle, the table structure change identification task can be initiated at midnight to reduce the impact on the execution of subsequent file scheduling tasks.

[0053] S602: Obtain a file configuration table.

[0054] Specifically, the information in the file configuration table can be read, such as the source system table name, the current column information of the downstream table, the downstream table structure change date, and the column information after the downstream table structure change.

[0055] S603: Whether the current batch date is earlier than the change date of the downstream table structure.

[0056] In some embodiments, the current batch date may be the current date of acquiring the file generation task. When the current batch date is earlier than the change date of the downstream table structure, it can be determined that the upstream table structure has not changed. Specifically, when the current date of acquiring the file generation task is not the change date of the downstream table structure, it can be said that the upstream table structure has not changed on the current date. Only when the upstream table structure has changed, the downstream table structure should also be changed accordingly.

[0057] In the case where the current date of acquiring the file generation task is earlier than the change date of the downstream table structure, S604 is executed.

[0058] S604: Obtain current column information of the upstream table from USER_TAB_COLUMNS.

[0059] USER_TAB_COLUMNS stores the information of the current user's tables, views and columns in Clusters, and is used by Oracle to obtain the table structure. The table structure change identification module 15 can obtain the current column information of the upstream table from USER_TAB_COLUMNS.

[0060] S605: Compare the current column information of the upstream table with the column information of the downstream table after the structure change, field by field.

[0061] S606: Whether the upstream table structure changes.

[0062] In some embodiments, the current column information of the upstream table and the column information of the downstream table after the structure change can be compared field by field. When the current column information of the upstream table is the same as the column information of the downstream table after the structure change, it is determined that the upstream table structure has not changed; when the current column information of the upstream table is different from the column information of the downstream table after the structure change, it is determined that the upstream table structure has changed.

[0063] If it is determined that the upstream table structure has changed, execute S607; otherwise, end the process.

[0064] S607: Modify the file configuration table and generate a log table.

[0065] Specifically, the file configuration table includes a field ISCHANGE indicating changes in the upstream table. The field ISCHANGE defaults to 0, indicating that the upstream table structure has not changed. If it is determined that the upstream table structure has changed, the value of the field ISCHANGE can be changed to 1 to indicate that the upstream table structure has changed.

[0066] In some embodiments, when it is determined that the upstream table structure has changed, the target field of the upstream table can be stored in the special processing field SPECIAL_COLS column of the file configuration table, and the SQL statement corresponding to the target field can be stored in the SQL statement SPECIAL_COLS_SQLS column corresponding to the special processing field of the file configuration table. The target field is the difference field between the current column information of the upstream table and the column information after the downstream table structure changes.

[0067] In some embodiments, when it is determined that the upstream table structure has changed, the difference fields between the current column information of the upstream table and the column information after the downstream table structure has changed can also be recorded in a log table, so as to update the downstream table structure according to the log table. Specifically, the log table can be sent to the user by email notification, so that the user can change the downstream table structure.

[0068] S502: Whether the current batch date is earlier than the change date of the downstream table structure.

[0069] If so, execute S503; otherwise, execute S504.

[0070] S503: Generate a file that complies with the downstream table structure according to the current column information of the downstream table.

[0071] When the current batch date is earlier than the change date of the downstream table structure, it can be determined that the upstream table structure has not changed. The upstream and downstream table structures should be consistent in theory, and the CURRENT_COLS and COLS column information are completely consistent. Therefore, a file that conforms to the downstream table structure can be generated based on the current column information CURRENT_COLS of the downstream table.

[0072] S504: Whether the upstream table structure has changed.

[0073] Specifically, since the table structure change identification module 15 has identified the table structure changes of the tables in the file configuration table one by one and updated the "ISCHANGE" field, it is possible to determine whether the upstream table structure has changed according to the "ISCHANGE" field in the file configuration table.

[0074] If so, execute S505; otherwise, execute S506.

[0075] S505: Generate a file that complies with the downstream table structure according to the current column information of the downstream table.

[0076] Specifically, when the upstream table structure changes, in order to make the generated file conform to the downstream table structure, the file can be generated according to the current column information of the downstream table. For table fields that need special processing, they can be processed according to "SPECIAL_COL_SQLS". For columns that do not exist in the first data table, they can be filled with null values.

[0077] S506: Generate a file that complies with the downstream table structure according to the current column information of the upstream table.

[0078] Specifically, when the upstream table structure has not changed, the upstream table structure is the same as the downstream table structure. The current column information of the upstream table can be directly obtained according to the USER_TAB_COLUMNS view to generate a file that conforms to the downstream table structure, so as to save a certain amount of field comparison time overhead.

[0079] S507: Whether the data file type is a fixed-length type.

[0080] If yes, execute S508; otherwise, execute 509.

[0081] S508: Generate a fixed-length format file.

[0082] S509: Generate a non-fixed-length format file.

[0083] S510: Update the completion status of the current batch task status and initialize the next batch task status.

[0084] It can be seen from the technical solutions provided in the above embodiments of this specification that in the embodiments of this specification, the vertical expansion of the system can be achieved through the file configuration table, making the expansion and maintenance of large-scale file generation tasks easy to achieve, ensuring that when the upstream table structure changes, the file format that adapts to the downstream table structure is provided in time, thereby improving the file generation efficiency.

[0085] See also Figure 7 . This specification embodiment also provides a file generation method. In the embodiment of this specification, the subject that executes the file generation method can be an electronic device with a logical operation function, and the electronic device can be a server. The server can be an electronic device with a certain computing and processing capability. It can have a network communication unit, a processor, a memory, etc. Of course, the server is not limited to the above-mentioned electronic device with a certain entity, and it can also be software running in the above-mentioned electronic device. The server can also be a distributed server, which can be a system with multiple processors, memories, network communication modules, etc. operating in collaboration. Alternatively, the server can also be a server cluster formed by several servers. The method can include the following steps.

[0086] S710: Obtain a file generation task; the file generation task includes converting target data in the first data table into a file that conforms to the structure of the second data table; the file generation task corresponds to a file configuration table; the file configuration table at least includes a change date of the second data table structure and current column information of the second data table.

[0087] In some embodiments, the first data table may be an upstream table, i.e., a data table storing upstream system data; the second data table may be a downstream table, i.e., a data table storing downstream system data. The file generation task includes converting the target data in the first data table into a file that conforms to the structure of the second data table.

[0088] In some embodiments, the file configuration table may include information about the upstream data table and the downstream data table involved in the file generation task, as shown in Table 1. The file configuration table may include information such as column information after the second data table structure changes, changes in the first data table structure, and file names. The file configuration table can be used to achieve vertical expansion of the system, making it easy to expand and maintain large batch file tasks.

[0089] In some embodiments, the change of the first data table structure in the file configuration table can be obtained in the following manner: when the current date of obtaining the file generation task is earlier than the change date of the second data table structure, it is determined that the first data table structure has not changed. Specifically, when the current date of obtaining the file generation task is not the change date of the second data table structure, it can be said that the first data table structure has not changed on the current date. Only when the first data table structure has changed, the second data table structure needs to be changed accordingly. By comparing the current date of the file generation task and the change date of the second data table structure, the change of the first data table structure can be quickly identified to improve the timeliness of task processing.

[0090] In some embodiments, the change of the first data table structure in the file configuration table can be obtained in the following manner: when the current date of acquiring the file generation task is later than or equal to the change date of the second data table structure, the current column information of the first data table is obtained; the current column information of the first data table and the column information after the change of the second data table structure are compared field by field, and when the current column information of the first data table is the same as the column information after the change of the second data table structure, it is determined that the first data table structure has not changed. By comparing the current column information of the first data table and the column information after the change of the second data table structure field by field, the first data table structure can be quickly identified, thereby improving the timeliness of task processing.

[0091] Specifically, the current column information of the first data table can be obtained from USER_TAB_COLUMNS, and the current column information of the first data table and the column information of the second data table after the structure change can be compared field by field. If the current column information of the first data table is the same as the column information of the second data table after the structure change, it is determined that the structure of the first data table has not changed; if the current column information of the first data table is different from the column information of the second data table after the structure change, it is determined that the structure of the first data table has changed. Obtaining the first data table structure information based on USER_TAB_COLUMNS to identify the first data table structure change can be extended to other relational databases, thereby improving the flexibility of file generation and arbitrary extension.

[0092] In some embodiments, the field ISCHANGE in the file configuration table can indicate the change of the first data table. The field ISCHANGE defaults to 0, indicating that the structure of the first data table has not changed. If it is determined that the structure of the first data table has changed, the value of the field ISCHANGE can be changed to 1 to indicate that the structure of the first data table has changed.

[0093] In some embodiments, when the current column information of the first data table is different from the column information after the structure of the second data table changes, the target field of the first data table is stored in the special processing field column of the file configuration table, and the sql statement corresponding to the target field is stored in the sql statement column corresponding to the special processing field of the file configuration table; wherein the target field is the difference field between the current column information of the first data table and the column information after the structure of the second data table changes. Specifically, when it is identified that the same field has changed, the field can be updated to the "SPECIAL_COLS" field, and the processing logic of the corresponding column is updated to the "SPECIAL_COL_SQLS" field. In the above manner, when it is identified that the same field has changed and requires special processing, the corresponding field can be automatically updated, thereby realizing flexible and efficient file generation that is adaptive to changes in upstream and downstream table structures.

[0094] In some embodiments, when the current column information of the first data table is different from the column information of the second data table after the structure is changed, the difference fields between the current column information of the first data table and the column information of the second data table after the structure is changed are recorded in a log table, so as to update the second data table structure according to the log table. Specifically, the log table can be sent to the user by email notification, so that the user can change the structure of the second data table. In the above manner, the user can be notified in time when the structure of the first data table changes, thereby improving the efficiency of emergency processing.

[0095] S720: When the current date of acquiring the file generation task is earlier than the change date of the second data table structure, convert the target data according to the current column information of the second data table to generate a file conforming to the second data table structure.

[0096] In some embodiments, when the current date for obtaining the file generation task is earlier than the change date of the second data table structure, the first data table structure and the second data table structure should theoretically be consistent, and the CURRENT_COLS and COLS column information are consistent, so a file that conforms to the second data table structure can be generated based on the current column information CURRENT_COLS of the second data table.

[0097] In some embodiments, when the current date of acquiring the file generation task is later than or equal to the change date of the second data table structure, if the first data table structure has changed, the target data is converted to generate a file that conforms to the second data table structure according to the column information of the changed second data table structure in the file configuration table. Specifically, for table fields that require special processing, they can be processed according to "SPECIAL_COL_SQLS". For columns that do not exist in the first data table, they can be filled with null values, thereby ensuring that when the first data table structure changes, a file that adapts to the second data table structure is accurately provided.

[0098] In some embodiments, when the acquisition date of the file generation task is later than or equal to the change date of the second data table structure, if the first data table structure has not changed, the target data is converted to generate a file that conforms to the second data table structure according to the current column information of the first data table. Specifically, when the first data table structure has not changed, the first data table structure is the same as the second data table structure, and the current column information of the first data table can be directly obtained according to the USER_TAB_COLUMNS view to generate a file that conforms to the second data table structure, so as to save a certain amount of field comparison time overhead and improve the efficiency of file generation.

[0099] It can be seen from the technical solutions provided in the above embodiments of this specification that, in the embodiments of this specification, a file generation task can be obtained; the file generation task includes converting the target data in the first data table into a file that conforms to the structure of the second data table; the file generation task corresponds to a file configuration table; the file configuration table includes at least the change date of the second data table structure and the current column information of the second data table; in the case where the current date of the file generation task is earlier than the change date of the second data table structure, the target data is converted to generate a file that conforms to the second data table structure according to the current column information of the second data table. The method provided in the embodiments of this specification can realize the vertical expansion of the system through the file configuration table, making the expansion and maintenance of large batch file generation tasks easy to achieve, ensuring that when the upstream table structure changes, the file format that adapts to the downstream table structure is provided in time, thereby improving the efficiency of file generation.

[0100] Figure 8 This is a functional structure diagram of an electronic device according to an embodiment of the present specification. The electronic device may include a memory and a processor.

[0101] In some embodiments, the memory can be used to store the computer program and / or module, and the processor implements various functions of the file generation method by running or executing the computer program and / or module stored in the memory, and calling the data stored in the memory. The memory may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system and an application required for at least one function; the data storage area may store data created according to the use of the user terminal. In addition, the memory may include a high-speed random access memory, and may also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0102] The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may be any conventional processor, etc. The processor may execute the computer instructions to implement the following steps: obtaining a file generation task; the file generation task includes converting target data in a first data table into a file conforming to a second data table structure; the file generation task corresponds to a file configuration table; the file configuration table includes at least a change date of the second data table structure and current column information of the second data table; when the current date of obtaining the file generation task is earlier than the change date of the second data table structure, converting the target data into a file conforming to the second data table structure according to the current column information of the second data table.

[0103] In the embodiments of this specification, the functions and effects specifically implemented by the electronic device can be explained in comparison with other embodiments and will not be repeated here.

[0104] Fig. 9 This is a functional structure diagram of a file generation device according to an embodiment of the present specification. The device may specifically include the following structural modules.

[0105] The acquisition module 910 is used to acquire a file generation task; the file generation task includes converting the target data in the first data table into a file conforming to the structure of the second data table; the file generation task corresponds to a file configuration table; the file configuration table at least includes a change date of the second data table structure and current column information of the second data table;

[0106] The generation module 920 is used to convert the target data into a file conforming to the second data table structure according to the current column information of the second data table when the current date of acquiring the file generation task is earlier than the change date of the second data table structure.

[0107] An embodiment of the present specification also provides a computer-readable storage medium of a file generation method, wherein the computer-readable storage medium stores computer program instructions, and when the computer program instructions are executed, the following are achieved: obtaining a file generation task; the file generation task includes converting target data in a first data table into a file that conforms to a second data table structure; the file generation task corresponds to a file configuration table; the file configuration table includes at least a change date of the second data table structure and current column information of the second data table; in a case where the current date of obtaining the file generation task is earlier than the change date of the second data table structure, converting the target data into a file that conforms to the second data table structure according to the current column information of the second data table.

[0108] In the embodiments of this specification, the above-mentioned storage medium includes but is not limited to random access memory (Random Access Memory, RAM), read-only memory (Read-Only Memory, ROM), cache (Cache), hard disk (Hard Disk Drive, HDD) or memory card (Memory Card). The memory can be used to store the computer program and / or module, and the memory can mainly include a storage program area and a storage data area, wherein the storage program area can store an operating system, at least one application required for a function, etc.; the storage data area can store data created according to the use of the user terminal, etc. In addition, the memory may include a high-speed random access memory and may also include a non-volatile memory. In the embodiments of this specification, the functions and effects specifically implemented by the program instructions stored in the computer-readable storage medium can be explained in comparison with other implementations, and will not be repeated here.

[0109] It should be noted that the file generation method, device and storage medium provided in the embodiments of this specification can be applied to the field of big data processing technology. Of course, it can also be applied to the financial field, or any field other than the financial field. The embodiments of this specification do not limit the application field of the file generation method, device and storage medium.

[0110] It should be noted that the various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device embodiment and the equipment embodiment, since they are basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0111] After reading this specification, those skilled in the art can conceive of any combination of some or all of the embodiments listed in this specification without creative work, and these combinations are also within the scope of disclosure and protection of this specification.

[0112] In the 1990s, improvements to a technology could be clearly distinguished as hardware improvements (for example, improvements to the circuit structure of diodes, transistors, switches, etc.) or software improvements (improvements to the method flow). However, with the development of technology, many improvements to the method flow today can be regarded as direct improvements to the hardware circuit structure. Designers almost always obtain the corresponding hardware circuit structure by programming the improved method flow into the hardware circuit. Therefore, it cannot be said that an improvement in a method flow cannot be implemented using a hardware entity module. For example, a programmable logic device (PLD) (such as a field programmable gate array (FPGA)) is such an integrated circuit whose logical function is determined by the user's programming of the device. Designers can "integrate" a digital system on a PLD by programming it themselves, without having to ask a chip manufacturer to design and produce a dedicated integrated circuit chip. Moreover, nowadays, instead of manually making integrated circuit chips, this kind of programming is mostly implemented by "logic compiler" software, which is similar to the software compiler used when developing and writing programs, and the original code before compilation must also be written in a specific programming language, which is called hardware description language (HDL). There is not only one HDL, but many kinds, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, RHDL (Ruby Hardware Description Language), etc. The most commonly used ones are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog2. Those skilled in the art should also know that it is only necessary to program the method flow slightly in the above-mentioned hardware description languages ​​and program it into the integrated circuit, and then it is easy to obtain the hardware circuit that implements the logic method flow.

[0113] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0114] It can be known from the above description of the implementation mode that the technicians in this field can clearly understand that the present specification can be implemented by means of software plus the necessary general hardware platform. Based on such an understanding, the technical solution of the present specification can be essentially or the part that contributes to the prior art can be embodied in the form of a software product, which can be stored in a storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present specification or some parts of the embodiments.

[0115] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0116] This specification can be used in many general or special computer system environments or configurations, such as personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronic devices, network PCs, minicomputers, mainframe computers, and distributed computing environments that include any of the above systems or devices.

[0117] This specification may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. This specification may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.

[0118] Although the present specification is described through embodiments, those skilled in the art will appreciate that there are many modifications and changes to the present specification without departing from the spirit of the present specification, and it is intended that the appended claims include these modifications and changes without departing from the spirit of the present specification.

Claims

1. A file generation method, characterized in that: The method comprises: Acquire a file generation task; the file generation task includes converting target data in a first data table into a file conforming to a second data table structure; the file generation task corresponds to a file configuration table; the file configuration table includes at least a change date of the second data table structure, current column information of the second data table, and column information after the second data table structure changes; Compare and obtain the current date of the file generation task and the change date of the second data table structure to identify the change of the first data table structure; In a case where the current date of acquiring the file generation task is earlier than the change date of the second data table structure, converting the target data into a file conforming to the second data table structure according to the current column information of the second data table; In the case where the current date of acquiring the file generation task is later than or equal to the change date of the second data table structure, if the first data table structure has changed, the target data is converted to generate a file conforming to the second data table structure according to the column information after the change of the second data table structure in the file configuration table.

2. The method according to claim 1, characterized in that The file configuration table also includes changes in the first data table structure.

3. The method according to claim 2, characterized in that The change of the first data table structure is obtained according to the following method: In a case where the current date of acquiring the file generation task is earlier than the change date of the second data table structure, it is determined that the first data table structure has not changed.

4. The method according to claim 2, characterized in that: The change of the first data table structure is obtained according to the following method: When the current date of acquiring the file generation task is later than or equal to the change date of the second data table structure, obtaining the current column information of the first data table; Compare the current column information of the first data table with the column information of the second data table after the structure change field by field, and when the current column information of the first data table is the same as the column information of the second data table after the structure change, determine that the structure of the first data table has not changed.

5. The method according to claim 4, characterized in that When the current column information of the first data table is different from the column information after the structure of the second data table is changed, it is determined that the structure of the first data table has been changed.

6. The method according to claim 5, characterized in that The method further comprises: When the current column information of the first data table is different from the column information of the second data table after the structure is changed, the target field of the first data table is stored in the special processing field column of the file configuration table, and the SQL statement corresponding to the target field is stored in the SQL statement column corresponding to the special processing field of the file configuration table; wherein the target field is the difference field between the current column information of the first data table and the column information of the second data table after the structure is changed.

7. The method according to claim 5, characterized in that The method further comprises: When the current column information of the first data table is different from the column information of the second data table after the structure is changed, the difference fields between the current column information of the first data table and the column information of the second data table after the structure is changed are recorded in a log table to facilitate updating the second data table structure according to the log table.

8. The method according to claim 1, characterized in that The method further comprises: When the acquisition date of the file generation task is later than or equal to the change date of the second data table structure, if the first data table structure has not changed, the target data is converted according to the current column information of the first data table to generate a file that conforms to the second data table structure.

9. A file generating device, characterized in that: The device comprises: An acquisition module is used to acquire a file generation task; the file generation task includes converting target data in a first data table into a file conforming to a second data table structure; the file generation task corresponds to a file configuration table; the file configuration table includes at least a change date of the second data table structure, current column information of the second data table, and column information after the second data table structure changes; A generation module is used to compare the current date of acquiring the file generation task and the change date of the second data table structure to identify the change of the first data table structure; when the current date of acquiring the file generation task is earlier than the change date of the second data table structure, convert the target data to generate a file that conforms to the second data table structure according to the current column information of the second data table; when the current date of acquiring the file generation task is later than or equal to the change date of the second data table structure, if the first data table structure has changed, convert the target data to generate a file that conforms to the second data table structure according to the column information of the changed second data table structure in the file configuration table.

10. An electronic device, characterized in that: include: Memory for storing computer programs; A processor is used to execute the computer program to achieve: obtaining a file generation task; The file generation task includes converting target data in a first data table into a file conforming to a second data table structure; the file generation task corresponds to a file configuration table; the file configuration table at least includes a change date of the second data table structure, current column information of the second data table, and column information after the change of the second data table structure; comparing and acquiring a current date of the file generation task and a change date of the second data table structure to identify changes in the first data table structure; in a case where the current date of the file generation task is earlier than the change date of the second data table structure, converting the target data into a file conforming to the second data table structure according to the current column information of the second data table; In the case where the current date of acquiring the file generation task is later than or equal to the change date of the second data table structure, if the first data table structure has changed, the target data is converted to generate a file conforming to the second data table structure according to the column information after the change of the second data table structure in the file configuration table.

11. A computer-readable storage medium, characterized in that: Computer instructions are stored thereon, and when the instructions are executed, the following are achieved: obtaining a file generation task; the file generation task includes converting target data in a first data table into a file conforming to a second data table structure; the file generation task corresponds to a file configuration table; the file configuration table includes at least a change date of the second data table structure, current column information of the second data table, and column information after the change of the second data table structure; comparing the current date of obtaining the file generation task and the change date of the second data table structure to identify the change of the first data table structure; in the case that the current date of obtaining the file generation task is earlier than the change date of the second data table structure, converting the target data into a file conforming to the second data table structure according to the current column information of the second data table; In the case where the current date of acquiring the file generation task is later than or equal to the change date of the second data table structure, if the first data table structure has changed, the target data is converted to generate a file conforming to the second data table structure according to the column information after the change of the second data table structure in the file configuration table.