File generation method and device, equipment, medium and program product

Abstract

The invention provides a file generation method which can be applied to the technical field of databases and the technical field of financial science and technology. The method comprises the steps of obtaining a data range of a to-be-generated file in response to starting of a file generation task about a source table in a system database; according to the data range, target data are obtained from multiple intermediate data tables related to the source table, data recorded by the multiple intermediate data tables are obtained from data sub-tables corresponding to the intermediate data tables, and the data sub-tables are obtained after fragmentation of the source table; according to the target data, generating intermediate files corresponding to the plurality of intermediate data tables to obtain a plurality of intermediate files; and combining the plurality of intermediate files to obtain a target file corresponding to the source table. The invention further provides a file generation device and equipment, a storage medium and a program product.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of database and the technical field of financial technology, in particular to a file generation method, device, equipment, medium and program product. BACKGROUND

[0002] A large amount of data is usually stored in a database, and in order to ensure the normal operation of the database, the data needs to be managed. As a common form of database management, file generation can show the data changes in the database by generating files of the database, so as to facilitate the management of the data in the database. For example, the generated file can directly show the position and time of a piece of data changed in the database, and can also show the deleted data in the database.

[0003] In the process of implementing the present application, the inventors have found that the file generation method in the related art has at least the following problems: the efficiency of the file generation method in the related art is low, and a large amount of memory resources are occupied. SUMMARY

[0004] In view of the above problems, the present application provides a file generation method, device, equipment, medium and program product.

[0005] According to a first aspect of the present application, a file generation method is provided, comprising: in response to the start of a file generation task for a source table in a system database, obtaining a data range of a file to be generated; obtaining target data from a plurality of intermediate data tables related to the source table according to the data range, wherein the data recorded in each of the plurality of intermediate data tables is obtained from a data sub-table corresponding to the intermediate data table, and the data sub-table is obtained by sharding the source table; generating an intermediate file corresponding to each of the plurality of intermediate data tables according to the target data, to obtain a plurality of intermediate files; and merging the plurality of intermediate files to obtain a target file corresponding to the source table.

[0006] According to an embodiment of the present application, the method further comprises: in response to the start of a data table update task for updating the intermediate data table, obtaining to-be-updated data from the data sub-table corresponding to the intermediate data table; determining a target storage mode of storing the to-be-updated data into the intermediate data table according to file generation information of the stored data in the intermediate data table; and storing the to-be-updated data according to the target storage mode.

[0007] According to an embodiment of the present application, the method further comprises: determining the target storage mode of the to-be-updated data stored in the intermediate data table according to the file generation information of the stored data in the intermediate data table, including: determining the target storage mode of the to-be-updated data stored in the intermediate data table as a deletion storage mode in a case where the file generation information indicates that the stored data has generated the intermediate file; and determining the target storage mode of the to-be-updated data stored in the intermediate data table as a direct storage mode in a case where the file generation information indicates that the stored data has not generated the intermediate file.

[0008] According to an embodiment of the present application, the method further comprises: obtaining the data range of the to-be-generated file in response to the start of the file generation task of the source table in the system database, including: obtaining file generation configuration content related to the file generation task of the source table in the system database in response to the start of the file generation task of the source table in the system database; and obtaining the data range of the to-be-generated file in a case where the file generation configuration content indicates that the file of the source table in the system database needs to be generated.

[0009] According to an embodiment of the present application, the method further comprises: recording state information of each of the plurality of intermediate files in a control table related to the plurality of intermediate files according to the intermediate file identifier information of each of the plurality of intermediate files.

[0010] The method further comprises: merging the plurality of intermediate files to obtain a target file corresponding to the source table in a case where the state information of each of the plurality of intermediate files in the control table indicates that the intermediate file is generated successfully.

[0011] According to an embodiment of the present application, the system database corresponds to a plurality of shard databases, and the data sub-tables of the source table obtained after being sharded according to a preset rule are stored in the corresponding shard databases.

[0012] The method further comprises: generating a data query statement corresponding to the shard database according to the data range and the preset rule; and obtaining the target data from the plurality of intermediate data tables related to the source table according to the data query statement.

[0013] According to an embodiment of the present application, the generation of the intermediate file corresponding to each of the plurality of intermediate data tables and the merging of the plurality of intermediate files are implemented through stream reading.

[0014] The second aspect of the present application provides a file generation apparatus, comprising: a first obtaining module configured to, in response to initiation of a file generation task for a source table in a system database, obtain a data range of a file to be generated; a second obtaining module configured to obtain target data from a plurality of intermediate data tables related to the source table according to the data range, wherein each of the plurality of intermediate data tables records data obtained from a data sub-table corresponding to the intermediate data table, and the data sub-table is obtained according to a fragmentation of the source table; a generating module configured to generate an intermediate file corresponding to each of the plurality of intermediate data tables according to the target data, and obtain a plurality of intermediate files; and an obtaining module configured to merge the plurality of intermediate files to obtain a target file corresponding to the source table.

[0015] The third aspect of the present application provides an electronic device, comprising: one or more processors; a memory configured to store one or more computer programs, wherein the one or more processors execute the one or more computer programs to implement the steps of the method.

[0016] The fourth aspect of the present application further provides a computer-readable storage medium having stored thereon a computer program or instructions, wherein the computer program or instructions, when executed by a processor, implement the steps of the method.

[0017] The fifth aspect of the present application further provides a computer program product comprising a computer program or instructions, wherein the computer program or instructions, when executed by a processor, implement the steps of the method.

[0018] According to the embodiments of the present application, in response to initiation of a file generation task for a source table in a system database, a data range of a file to be generated is obtained; target data is obtained from a plurality of intermediate data tables related to the source table according to the data range, wherein each of the plurality of intermediate data tables records data obtained from a data sub-table corresponding to the intermediate data table, and the data sub-table is obtained according to a fragmentation of the source table; an intermediate file corresponding to each of the plurality of intermediate data tables is generated according to the target data, and a plurality of intermediate files are obtained; and the plurality of intermediate files are merged to obtain a target file corresponding to the source table. Since the source table in the database is fragmented into a plurality of data sub-tables and is set in the intermediate data tables corresponding to the data sub-tables, the process of generating a file by querying data from the source table is converted into obtaining data from the plurality of data sub-tables and storing the data in the intermediate data tables, and then obtaining target data from the intermediate data tables according to the data range of the file to be generated, thereby reducing the data query pressure of the system database and improving the efficiency of file generation. BRIEF DESCRIPTION OF DRAWINGS

[0019] The above content of the present application and other purposes, features and advantages will be more apparent through the following description of the embodiments of the present application with reference to the accompanying drawings, in which:

[0020] Figure 1 An application scenario diagram of a file generation method, apparatus, device, medium and program product according to an embodiment of the present application is schematically shown;

[0021] Figure 2 A flowchart of a file generation method according to an embodiment of the present application is schematically shown;

[0022] Figure 3 A flowchart of a file generation method according to another embodiment of the present application is schematically shown;

[0023] Figure 4 A data flowchart of a file generation method in a system database according to an embodiment of the present application is schematically shown;

[0024] Figure 5 A structural block diagram of a file generation apparatus according to an embodiment of the present application is schematically shown; and

[0025] Figure 6 A block diagram of an electronic device suitable for implementing a file generation method according to an embodiment of the present application is schematically shown. DETAILED DESCRIPTION

[0026] Hereinafter, embodiments of the present application will be described with reference to the accompanying drawings. It should be understood, however, that the description which follows is merely exemplary and is not intended to limit the scope of the application. In the following detailed description of embodiments of the present application, numerous specific details are set forth in order to provide a thorough understanding of the present application. However, it will be apparent to one skilled in the art that one or more embodiments of the present application can be practiced without these specific details. In other instances, well-known structures and functions have not been described in detail in order to avoid obscuring aspects of the present application.

[0027] The terms used herein are merely used to describe specific embodiments and are not intended to limit the present application. The terms "include", "comprise" and the like used herein indicate the presence of the described features, steps, operations and / or components, but do not exclude the presence or addition of one or more other features, steps, operations or components.

[0028] All terms used herein (including technical and scientific terms) have meanings commonly understood by one of ordinary skill in the art, unless otherwise defined. It should be noted that the terms used herein should be interpreted as having meanings consistent with the context of the specification, and should not be interpreted in an idealized or overly formal manner.

[0029] When using expressions such as "at least one of A, B and C", they should generally be interpreted in accordance with the meaning that is commonly understood by those skilled in the art (e.g., "a system having at least one of A, B and C" should include, but is not limited to, a system having A alone, a system having B alone, a system having C alone, a system having A and B, a system having A and C, a system having B and C, and / or a system having A, B and C, etc.).

[0030] In the technical solution of this application, the user information (including but not limited to user personal information, user image information, user device information, such as location information) and data (including but not limited to data used for analysis, stored data, and displayed data) involved are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, storage, use, processing, transmission, provision, disclosure, and application of related data all comply with relevant laws, regulations, and standards, take necessary confidentiality measures, do not violate public order and good morals, and provide corresponding operation entry points for users to choose to authorize or refuse.

[0031] In scenarios involving automated decision-making using personal information, the methods, devices, and systems provided in this application all offer users corresponding entry points for choosing to agree to or reject the automated decision-making results. If the user chooses to reject, the process proceeds to the expert decision-making stage. Here, "automated decision-making" refers to the activity of automatically analyzing and evaluating an individual's behavioral habits, interests, or economic, health, and credit status through computer programs, and then making a decision. Here, "expert decision-making" refers to the activity of making decisions by personnel who specialize in a particular field, possess specialized experience, knowledge, and skills, and have reached a certain level of professional expertise.

[0032] File generation typically involves retrieving data from the database in one go, storing it in memory, and then writing it to the file. However, as the amount of data in the database increases, the query pressure on a single database also gradually increases. Retrieving the required data from a specific source table in the database results in a significant decrease in query speed. Directly retrieving data from the database and then writing it to the file would consume even more memory, potentially causing server crashes and impacting production system operations. Therefore, this method is not suitable for databases with large query volumes.

[0033] In view of this, embodiments of this application provide a file generation method, including: in response to the initiation of a file generation task about a source table in a system database, obtaining the data range of the file to be generated; obtaining target data from multiple intermediate data tables related to the source table according to the data range, wherein the data recorded in each of the multiple intermediate data tables is obtained from a data partition table corresponding to the intermediate data table, and the data partition table is obtained after partitioning the source table; generating intermediate files corresponding to each of the multiple intermediate data tables according to the target data, thereby obtaining multiple intermediate files; and merging the multiple intermediate files to obtain a target file corresponding to the source table.

[0034] Figure 1 The illustrations depict application scenarios of document generation methods, apparatus, devices, media, and program products according to embodiments of this application.

[0035] like Figure 1 As shown, application scenario 100 according to this embodiment may include a first terminal device 101, a second terminal device 102, a third terminal device 103, a network 104, and a server 105. The network 104 serves as a medium for providing a communication link between the first terminal device 101, the second terminal device 102, the third terminal device 103, and the server 105. The network 104 may include various connection types, such as wired or wireless communication links, or fiber optic cables, etc.

[0036] Users can use the first terminal device 101, the second terminal device 102, and the third terminal device 103 to interact with the server 105 via the network 104 to receive or send messages, etc. Various communication client applications can be installed on the first terminal device 101, the second terminal device 102, and the third terminal device 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social media platform software, etc. (for example only).

[0037] The first terminal device 101, the second terminal device 102, and the third terminal device 103 can be various electronic devices with displays and support web browsing, including but not limited to smartphones, tablets, laptops, and desktop computers.

[0038] Server 105 can be a server that provides various services, such as a backend management server that supports websites browsed by users using the first terminal device 101, the second terminal device 102, and the third terminal device 103 (this is just an example). The backend management server can analyze and process data such as received user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal devices.

[0039] It should be noted that the file generation method provided in this application embodiment can generally be executed by server 105. Correspondingly, the file generation apparatus provided in this application embodiment can generally be located in server 105. The file generation method provided in this application embodiment can also be executed by a server or server cluster that is different from server 105 and capable of communicating with the first terminal device 101, the second terminal device 102, the third terminal device 103, and / or server 105. Correspondingly, the file generation apparatus provided in this application embodiment can also be located in a server or server cluster that is different from server 105 and capable of communicating with the first terminal device 101, the second terminal device 102, the third terminal device 103, and / or server 105.

[0040] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0041] Figure 2 A flowchart illustrating a document generation method according to an embodiment of this application is shown schematically.

[0042] like Figure 2 As shown, the file generation method of this embodiment includes operations S210 to S240.

[0043] In operation S210, in response to the initiation of a file generation task for a source table in the system database, the data range of the file to be generated is obtained.

[0044] In operation S220, target data is obtained from multiple intermediate data tables related to the source table according to the data range. The data recorded in each of the multiple intermediate data tables is obtained from the data partition table corresponding to the intermediate data table. The data partition table is obtained after the source table is partitioned.

[0045] In operation S230, based on the target data, intermediate files corresponding to multiple intermediate data tables are generated, resulting in multiple intermediate files.

[0046] In operation S240, multiple intermediate files are merged to obtain the target file corresponding to the source table.

[0047] According to embodiments of this application, the system database can be, for example, a database for a transaction system, user system, or product system in the financial industry. The source tables in the system database can be data tables that store transaction data, user change data, and product data. The database may include multiple source tables.

[0048] According to embodiments of this application, a timed file generation task can be set. When the file generation task starts, the data range of the file to be generated is obtained. The data range of the file to be generated can be, for example, the data of the current day. The data range of the file to be generated can be obtained from the system database when the file generation task starts. The data range of the file to be generated can also be pre-stored in the system database or server and can be obtained from the system database or server when the file generation task starts.

[0049] According to embodiments of this application, the source table of a database can be sharded to obtain multiple data shards, thereby distributing the data volume in the source table across multiple data shards. Each data shard can also be used to create an intermediate data table, which retrieves data from the corresponding data shard. Target data can be retrieved from the intermediate data table based on a data range.

[0050] According to an embodiment of this application, since the target data is obtained from an intermediate data table, an intermediate file corresponding to the intermediate data table can be generated from the data in the target data that corresponds to the intermediate data table, thereby obtaining multiple intermediate files.

[0051] According to embodiments of this application, multiple intermediate files are generated based on target data, which is the data range of the file to be generated. Therefore, merging multiple intermediate files yields a target file corresponding to the source table.

[0052] According to embodiments of this application, in response to the initiation of a file generation task for a source table in a system database, the data range of the file to be generated is obtained; target data is obtained from multiple intermediate data tables related to the source table according to the data range, wherein the data recorded in each of the multiple intermediate data tables is obtained from a data partition table corresponding to the intermediate data table, and the data partition table is obtained after sharding the source table; intermediate files corresponding to each of the multiple intermediate data tables are generated according to the target data, resulting in multiple intermediate files; the multiple intermediate files are merged to obtain the target file corresponding to the source table. Since the source table in the database is sharded into multiple data partition tables and intermediate data tables are set corresponding to the data partition tables, the original process of querying data from the source table to generate a file is transformed into obtaining data from multiple data partition tables, storing it in intermediate data tables, and then obtaining the target data from the intermediate data tables according to the data range of the file to be generated. This reduces the data query pressure on the system database and improves the efficiency of file generation.

[0053] According to an embodiment of this application, in response to the initiation of a file generation task for a source table in the system database, obtaining the data range of the file to be generated may include: in response to the initiation of a file generation task for a source table in the system database, obtaining file generation configuration content related to the file generation task; and, if it is determined that the file generation configuration content represents the need to generate a file for a source table in the system database, obtaining the data range of the file to be generated.

[0054] According to embodiments of this application, the file generation requirement can be set according to actual needs. For example, data generated on weekdays needs to be generated into corresponding files every day, while data generated on rest days can be generated into files together with the first workday after the rest day. Therefore, file generation configuration content can be set. After the file generation task starts, the file generation configuration content is obtained. If the file generation configuration content indicates that a file needs to be generated, the data range of the file to be generated is obtained. If the file generation configuration content indicates that a file does not need to be generated, no subsequent operations are performed.

[0055] According to an embodiment of this application, the file generation configuration content may include a file generation identifier. When the value of the file generation identifier is 1, it indicates that a file needs to be generated. When the value of the file generation identifier is 0, it indicates that a file does not need to be generated.

[0056] According to embodiments of this application, by setting file generation configuration content, different situations of file generation can be flexibly configured, thereby improving the flexibility of the file generation method.

[0057] Figure 3 A flowchart illustrating a document generation method according to yet another embodiment of this application is shown.

[0058] like Figure 3 As shown, the file generation method of this embodiment includes operations S310 to S330.

[0059] In operation S310, in response to the initiation of a data table update task for updating an intermediate data table, the data to be updated is retrieved from the data sub-table corresponding to the intermediate data table.

[0060] In operation S320, based on the file generation information of the existing data in the intermediate data table, the target storage method for storing the data to be updated to the intermediate data table is determined.

[0061] When operating S330, store the data to be updated according to the target storage method.

[0062] According to embodiments of this application, the data table update task can be a scheduled task, for example, started daily. Updates to intermediate data tables can be performed based on the initiation of the data table update task.

[0063] According to an embodiment of this application, when a data table update task is started, that is, when the intermediate data table needs to be updated, the data to be updated can be obtained from the data sub-table corresponding to the intermediate data table. The data to be updated can be the changed data in the data sub-table after the last data table update task was started. Therefore, a time condition can be set to query the data between the start time of the last data table update task and the start time of this data table update task from the data sub-table, thereby determining the data to be updated.

[0064] According to the embodiments of this application, the intermediate data table can record existing data. The file generation information of the existing data can indicate whether the existing data has been generated into a file. Based on whether the existing data has been generated into a file, different storage methods for the data to be updated can be determined. Therefore, based on the file generation information, the target storage method for storing the data to be updated into the intermediate data table can be determined, and the data to be updated can be stored according to the target storage method.

[0065] According to the embodiments of this application, when the data to be updated is obtained from the data partition table and stored in the intermediate data table, logical processing can also be performed on the data to be updated, such as field value regular expression matching and replacement, nested subqueries, dictionary mapping, etc. The data in the intermediate data table is the processed data, which can be directly used for file generation or as source data for other special logical processing.

[0066] According to embodiments of this application, the generation of the intermediate data table and the file generation can be independent of each other. When the file is not being generated, the intermediate data table can still obtain data. Therefore, the intermediate data table can store the historical records of the data partition table, avoiding the data of the data partition table being overwritten when the file is not being used, thus avoiding the need for complex processing logic to query the overwritten data.

[0067] According to an embodiment of this application, determining the target storage method for storing the data to be updated in the intermediate data table based on the file generation information of the existing data in the intermediate data table may include: if the file generation information indicates that the existing data has generated an intermediate file, determining the target storage method for storing the data to be updated in the intermediate data table as a deletion storage method; if the file generation information indicates that the existing data has not generated an intermediate file, determining the storage method for storing the data to be updated in the intermediate data table as a direct storage method.

[0068] According to the embodiments of this application, when it is determined that the file generation information indicates that the existing data has been generated into an intermediate file, that is, the existing data has completed the file generation and does not need to be generated again, the target storage method for storing the data to be updated into the intermediate data table can be determined to be the deletion storage method. The deletion storage method indicates that the existing data is deleted from the intermediate data table and then the data to be updated is stored into the intermediate data table.

[0069] According to the embodiments of this application, when it is determined that the file generation information indicates that the existing data has not yet generated an intermediate file, that is, the existing data has not yet completed file generation and needs to be generated, it can be determined that the storage method of storing the data to be updated in the intermediate data table is the direct storage method, which means that the data to be updated is directly stored in the intermediate data table.

[0070] According to the embodiments of this application, the storage method of the data to be updated in the intermediate data table is determined based on the different contents represented by the file generation information of the existing data. This can avoid storing too much data in the intermediate data table and reduce the data complexity in the intermediate data table.

[0071] According to an embodiment of this application, the file generation method may further include: recording the status information of each of the multiple intermediate files in a control table associated with the multiple intermediate files based on the intermediate file identification information of each of the multiple intermediate files.

[0072] According to embodiments of this application, the control table can record the status information of intermediate files, such as pending generation, generation in progress, generation successful, generation failed, etc. The control table can also record intermediate file identification information, for example, intermediate file identification information could be intermediate file A1, intermediate file A2, and intermediate file A3. Based on the intermediate file identification information, the status information of multiple intermediate files is recorded in the control table for their respective purposes, enabling the management of intermediate files.

[0073] According to an embodiment of this application, merging multiple intermediate files to obtain a target file corresponding to the source table may include: merging multiple intermediate files to obtain a target file corresponding to the source table when the status information of each of the multiple intermediate files in the control table indicates that the intermediate file has been successfully generated.

[0074] According to the embodiments of this application, the status information of multiple intermediate files can be queried in the control table based on the intermediate file identification information of the intermediate files. If the status information of multiple intermediate files in the control table indicates that the intermediate files have been successfully generated, the multiple intermediate files can be merged to obtain the target file corresponding to the source table.

[0075] According to embodiments of this application, the status information of intermediate files is recorded through a control table, which facilitates a quick understanding of the status of intermediate files, thereby improving the generation efficiency of target files.

[0076] According to the embodiments of this application, the system database corresponds to multiple sharded databases, and the data obtained after the source table is sharded according to preset rules is stored in the corresponding sharded database.

[0077] According to embodiments of this application, preset rules may include data region, data type, etc.

[0078] According to embodiments of this application, the system database is divided into multiple sharded databases, and data is stored in tables in the corresponding sharded databases, which can alleviate the query pressure on the system database. The sharded data can be a separate physical machine or a virtual machine.

[0079] According to an embodiment of this application, obtaining target data from multiple intermediate data tables related to the source table based on the data range may include: generating a data query statement corresponding to the sharded database based on the data range and preset rules; and obtaining the target data from multiple intermediate data tables related to the source table based on the data query statement.

[0080] According to the embodiments of this application, the corresponding sharded database is determined based on the data range, and the data sharding rules corresponding to the sharded database are determined according to preset rules, thereby avoiding the generation of data query statements corresponding to the sharded database.

[0081] According to the embodiments of this application, there may be a primary and backup situation in the sharded databases. For example, sharded database A is the primary database and sharded database B is the backup database corresponding to sharded database A. Therefore, generating a data query statement corresponding to the sharded data and obtaining the target data according to the data query statement can avoid querying duplicate data in the primary and backup sharded databases, which would lead to inaccurate data.

[0082] According to embodiments of this application, the generation of intermediate files corresponding to each of the multiple intermediate data tables and the merging of the multiple intermediate files are achieved through streaming reading.

[0083] According to the embodiments of this application, intermediate files are generated and merged through streaming reading, which allows for simultaneous reading and writing, improving file generation efficiency while avoiding excessive server memory consumption.

[0084] Figure 4 The diagram illustrates a data flow diagram in the system database of the file generation method according to an embodiment of this application.

[0085] like Figure 4As shown, the system database can adopt a multi-shard model, splitting the data in the original single system database into multiple parts according to preset rules (such as uniform or combined fields with clear distinguishing identifiers) and storing them in different shard databases, thus distributing the system database load. The system database can correspond to n shard databases, such as shard database 1, shard database 2, ..., shard database n. The system database can include source table A, source table B, ..., source table T. Source table A can be divided into data shard tables A1, A2, ..., An; source table B can be divided into data shard tables B1, B2, ..., Bn; ...; source table T can be divided into data shard tables T1, T2, ..., Tn. Data shards A1, B1, ..., T1 can be stored in sharded database 1; data shards A2, B2, ..., T2 can be stored in sharded database 2; ..., data shards An, Bn, ..., Tn can be stored in sharded database n. Shard databases can be dedicated physical machines or virtual machines. After the source table data is sharded, if new business data is added, it will be automatically routed to the corresponding shard database according to the sharding rules, ensuring physical data isolation between multiple shard databases and providing a foundation for subsequent file generation.

[0086] According to embodiments of this application, the file generation pattern can be determined based on the needs of the requesting party. For example, the file generation pattern may include generating files daily on weekdays, and generating files together with data from rest days and the first workday after the rest day. Therefore, to meet the specific needs of rest days and avoid potentially complex data processing procedures caused by rest day data being overwritten on workdays, an intermediate data table is designed. Figure 4 As shown, data table A1 corresponds to intermediate data table A1, data table B1 corresponds to intermediate data table B1, data table T1 corresponds to intermediate data table T1, data table A2 corresponds to intermediate data table A2, data table B2 corresponds to intermediate data table B2, data table T2 corresponds to intermediate data table T2, data table A3 corresponds to intermediate data table A3, data table B3 corresponds to intermediate data table B3, and data table T3 corresponds to intermediate data table T3. Each intermediate data table retrieves the data to be updated from its corresponding data table. During the process of storing the data to be updated in the intermediate data table, simple processing operations, such as field changes, can also be performed on the data to be updated.

[0087] According to the embodiments of this application, when the source table data is directly exported, it is difficult to write the query logic if a real-time query method is used. Furthermore, the data in the source table has already been updated to the latest state and has no historical records. However, the intermediate data table has a smaller data volume and can retain records of recent historical states (such as holiday data). If there are special needs for these records, they can still be further processed.

[0088] According to the embodiments of this application, storing recent historical data in an intermediate data table can meet the specific business needs of the regulatory authorities while avoiding the complex processing logic (such as field value regular expression matching and replacement, nested subqueries, dictionary mapping, etc.) that may be involved in directly querying the business source table. The intermediate data table can not only track recent changes in data by storing the historical state of the data to achieve the purpose of data supervision, but also avoid the problem of low efficiency in obtaining data by directly querying the source table. To further improve data efficiency, some logical processing such as field value regular expression matching and replacement, nested subqueries, and dictionary mapping can be completed in advance when retrieving data from the source table. In other words, the data in the intermediate data table is the processed data, which can be directly used for file generation or as source data for other special logical processing.

[0089] According to embodiments of this application, file generation tasks and data table update tasks can be independent tasks. Since the sharded databases have achieved physical data isolation, file generation tasks on each sharded database can be started simultaneously for parallel processing. This transforms individual queries to the source table into queries to intermediate data tables in each sharded database, reducing the amount of data in the intermediate data tables compared to the source table. Therefore, query speed is improved, resulting in increased query efficiency and reduced pressure on the system database. The data query statements for the intermediate data tables can be determined based on the data range and preset rules, avoiding the problem of sharded databases acting as primary / backup entities, which could lead to inaccurate query results. When the file generation task is started, the client's requirements can be obtained, and files can be accurately generated based on those requirements, avoiding the generation of redundant files.

[0090] According to embodiments of this application, file generation configuration content can be obtained during the file generation process. The decision on whether to actually generate a file in the daily file generation task is made based on this configuration content. If so, the file export query only needs to query the intermediate data table, without needing to query the source table again. Furthermore, during the process of exporting target data from the intermediate data table to generate a file, streaming reading can be used, reading and writing simultaneously. This improves file generation efficiency while avoiding excessive server memory consumption. If the query result is indeed empty, a corresponding empty file can be generated (it should be noted that the target path for file generation here is a shared path with sufficient bandwidth). Figure 4As shown, the data corresponding to intermediate data table A1 generates intermediate file A1, the data corresponding to intermediate data table B1 generates intermediate file B1, the data corresponding to intermediate data table T1 generates intermediate file T1, the data corresponding to A2 generates intermediate file A2, the data corresponding to intermediate data table B2 generates intermediate file B2, the data corresponding to intermediate data table T2 generates intermediate file T2, the data corresponding to An generates intermediate file An, the data corresponding to intermediate data table Bn generates intermediate file Bn, and the data corresponding to intermediate data table Tn generates intermediate file Tn.

[0091] According to the embodiments of this application, before merging files, the file generation identifier provided by the regulatory end can be determined. If the file generation identifier indicates that a file needs to be generated, subsequent operations can continue. If the file generation identifier indicates that a file does not need to be generated, subsequent operations can be skipped, and no file generation is required. If file generation is required, the control table can be used to determine whether each intermediate file has been successfully generated. If all intermediate files have been successfully generated, the intermediate files can be merged. Intermediate files A1, A2, ..., An can be merged to obtain intermediate file A; intermediate files B1, B2, ..., Bn can be merged to obtain intermediate file B; and intermediate files T1, T2, ..., Tn can be merged to obtain intermediate file T. The file merging process also uses streaming reading to avoid excessive server memory consumption.

[0092] According to the embodiments of this application, in the multi-sharding mode, for an intermediate data table in a single sharded database, it can still be further split into multiple intermediate data sub-tables or a multi-threaded approach can be adopted to further improve processing efficiency.

[0093] According to the embodiments of this application, the number of source tables and the number of sharded databases listed in the figures are for illustrative purposes only, and other numbers of source tables and sharded databases may be set as needed.

[0094] According to the embodiments of this application, since the source table in the database is sharded into multiple data shards and intermediate data tables are set in the corresponding data shards, the original process of querying data from the source table to generate a file is transformed into obtaining data from multiple data shards and storing it in the intermediate data table, and then obtaining the target data from the intermediate data table according to the data range of the file to be generated, thereby reducing the data query pressure on the system database and improving the efficiency of file generation.

[0095] Based on the above-described document generation method, this application also provides a document generation apparatus. The following will be combined with... Figure 5 The device is described in detail.

[0096] Figure 5 A schematic block diagram of a document generation apparatus according to an embodiment of this application is shown.

[0097] like Figure 5 As shown, the file generation device 500 of this embodiment includes a first acquisition module 510, a second acquisition module 520, a generation module 530, and a obtaining module 540.

[0098] The first acquisition module 510 is used to acquire the data range of the file to be generated in response to the initiation of a file generation task for the source table in the system database. In one embodiment, the first acquisition module 510 can be used to perform the operation S210 described above, which will not be repeated here.

[0099] The second acquisition module 520 is used to acquire target data from multiple intermediate data tables related to the source table according to the data range. The data recorded in each of the multiple intermediate data tables is obtained from a corresponding data partition table, which is obtained by partitioning the source table. In one embodiment, the second acquisition module 520 can be used to execute the operation S220 described above, which will not be repeated here.

[0100] The generation module 530 is used to generate intermediate files corresponding to each of the multiple intermediate data tables based on the target data, thereby obtaining multiple intermediate files. In one embodiment, the generation module 530 can be used to perform the operation S230 described above, which will not be repeated here.

[0101] The module 540 is used to merge multiple intermediate files to obtain the target file corresponding to the source table. In one embodiment, the module 540 can be used to perform the operation S240 described above, which will not be repeated here.

[0102] According to an embodiment of this application, the apparatus 500 further includes: a third acquisition module, configured to acquire data to be updated from a data sub-table corresponding to the intermediate data table in response to the initiation of a data table update task for updating the intermediate data table; a determination module, configured to determine a target storage method for storing the data to be updated in the intermediate data table based on file generation information of the data already stored in the intermediate data table; and a storage module, configured to store the data to be updated according to the target storage method.

[0103] According to an embodiment of this application, a determining module for determining the target storage method of storing data to be updated in an intermediate data table based on file generation information of existing data in an intermediate data table includes: a first determining unit, used to determine that the target storage method of storing data to be updated in an intermediate data table is a deletion storage method when the file generation information indicates that the existing data has generated an intermediate file; and a second determining unit, used to determine that the storage method of storing data to be updated in an intermediate data table is a direct storage method when the file generation information indicates that the existing data has not generated an intermediate file.

[0104] According to an embodiment of this application, a first acquisition module 510 for acquiring the data range of a file to be generated in response to the initiation of a file generation task about a source table in a system database includes: a first acquisition unit for acquiring file generation configuration content related to the file generation task in response to the initiation of a file generation task about a source table in a system database; and a second acquisition unit for acquiring the data range of the file to be generated when it is determined that the file generation configuration content indicates that a file of a source table in a system database needs to be generated.

[0105] According to an embodiment of this application, the device 500 further includes: a recording module, configured to record the status information of each of the multiple intermediate files in a control table associated with the multiple intermediate files based on the intermediate file identification information of each of the multiple intermediate files.

[0106] According to an embodiment of this application, a module 540 for merging multiple intermediate files to obtain a target file corresponding to a source table includes: a unit for merging multiple intermediate files to obtain a target file corresponding to a source table when the status information of each of the multiple intermediate files in the control table indicates that the intermediate file has been successfully generated.

[0107] According to the embodiments of this application, the system database corresponds to multiple sharded databases, and the data obtained after the source table is sharded according to preset rules is stored in the corresponding sharded database.

[0108] According to an embodiment of this application, a second acquisition module 520 for obtaining target data from multiple intermediate data tables related to a source table based on a data range includes: a third acquisition unit for generating a data query statement corresponding to a sharded database based on a data range and preset rules; and a fourth acquisition unit for obtaining target data from multiple intermediate data tables related to a source table based on the data query statement.

[0109] According to embodiments of this application, the generation of intermediate files corresponding to each of the multiple intermediate data tables and the merging of the multiple intermediate files are achieved through streaming reading.

[0110] According to embodiments of this application, any plurality of modules among the first acquisition module 510, the second acquisition module 520, the generation module 530, and the obtaining module 540 can be combined into one module, or any one of these modules can be split into multiple modules. Alternatively, at least part of the functionality of one or more of these modules can be combined with at least part of the functionality of other modules and implemented in one module. According to embodiments of this application, at least one of the first acquisition module 510, the second acquisition module 520, the generation module 530, and the obtaining module 540 can be at least partially implemented as hardware circuitry, such as a field-programmable gate array (FPGA), a programmable logic array (PLA), a system-on-a-chip, a system-on-a-substrate, a system-on-package, an application-specific integrated circuit (ASIC), or implemented in hardware or firmware by any other reasonable means of integrating or packaging the circuitry, or implemented in any one of the three implementation methods of software, hardware, and firmware, or in a suitable combination of any of these. Alternatively, at least one of the first acquisition module 510, the second acquisition module 520, the generation module 530, and the obtaining module 540 may be implemented at least partially as a computer program module, which can perform corresponding functions when the computer program module is run.

[0111] Figure 6 A block diagram schematically illustrates an electronic device suitable for implementing a document generation method according to an embodiment of this application.

[0112] like Figure 6 As shown, an electronic device 600 according to an embodiment of this application includes a processor 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage portion 608 into a random access memory (RAM) 603. The processor 601 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or an associated chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 601 may also include onboard memory for caching purposes. The processor 601 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of this application.

[0113] RAM 603 stores various programs and data required for the operation of electronic device 600. Processor 601, ROM 602, and RAM 603 are interconnected via bus 604. Processor 601 executes various operations of the method flow according to embodiments of this application by executing programs in ROM 602 and / or RAM 603. It should be noted that the programs may also be stored in one or more memories other than ROM 602 and RAM 603. Processor 601 may also execute various operations of the method flow according to embodiments of this application by executing programs stored in said one or more memories.

[0114] According to embodiments of this application, the electronic device 600 may further include an input / output (I / O) interface 605, which is also connected to a bus 604. The electronic device 600 may also include one or more of the following components connected to the input / output (I / O) interface 605: an input section 606 including a keyboard, mouse, etc.; an output section 607 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 608 including a hard disk, etc.; and a communication section 609 including a network interface card such as a LAN card, modem, etc. The communication section 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to the input / output (I / O) interface 605 as needed. A removable medium 611, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 610 as needed so that computer programs read from it can be installed into the storage section 608 as needed.

[0115] This application also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or it may exist independently and not assembled into the device / apparatus / system. The computer-readable storage medium carries one or more programs, which, when executed, implement the method according to the embodiments of this application.

[0116] According to embodiments of this application, the computer-readable storage medium can be a non-volatile computer-readable storage medium, such as including but not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this application, the computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. For example, according to embodiments of this application, the computer-readable storage medium may include ROM 602 and / or RAM 603 and / or one or more memories other than ROM 602 and RAM 603 described above.

[0117] Embodiments of this application also include a computer program product comprising a computer program containing program code for performing the methods shown in the flowchart. When the computer program product is run on a computer system, the program code is used to cause the computer system to implement the methods provided in the embodiments of this application.

[0118] When the computer program is executed by the processor 601, it performs the functions defined in the system / apparatus of this application embodiment. According to the embodiments of this application, the systems, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0119] In one embodiment, the computer program may rely on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may also be transmitted and distributed in the form of signals over a network medium, and downloaded and installed via the communication section 609, and / or installed from the removable medium 611. The program code contained in the computer program can be transmitted using any suitable network medium, including but not limited to: wireless, wired, etc., or any suitable combination thereof.

[0120] In such an embodiment, the computer program can be downloaded and installed from a network via the communication section 609, and / or installed from the removable medium 611. When the computer program is executed by the processor 601, it performs the functions defined in the system of this application embodiment. According to the embodiments of this application, the systems, devices, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0121] According to embodiments of this application, program code for executing the computer programs provided in the embodiments of this application can be written in any combination of one or more programming languages. Specifically, these computational programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, languages ​​such as Java, C++, Python, "C", or similar programming languages. The program code can be executed entirely on the user's computing device, partially on the user's device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0122] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0123] Those skilled in the art will understand that the features described in the various embodiments of this application can be combined and / or combined in various ways, even if such combinations or combinations are not explicitly described in this application. In particular, the features described in the various embodiments of this application can be combined and / or combined in various ways without departing from the spirit and teachings of this application. All such combinations and / or combinations fall within the scope of this application.

Claims

1. A method for generating a file, characterized in that, The method includes: In response to the initiation of a file generation task for a source table in the system database, obtain the data range of the file to be generated; Target data is obtained from multiple intermediate data tables related to the source table according to the data range, wherein the data recorded in each of the multiple intermediate data tables is obtained from the data partition table corresponding to the intermediate table, and the data partition table is obtained after the source table is partitioned; Based on the target data, intermediate files corresponding to each of the multiple intermediate data tables are generated, resulting in multiple intermediate files; Merge multiple intermediate files to obtain the target file corresponding to the source table.

2. The method according to claim 1, characterized in that, The method further includes: In response to the initiation of a data table update task for updating the intermediate data table, the data to be updated is obtained from the data sub-table corresponding to the intermediate data table; Based on the file generation information of the existing data in the intermediate data table, determine the target storage method for storing the data to be updated in the intermediate data table; The data to be updated is stored according to the target storage method.

3. The method according to claim 2, characterized in that, The step of determining the target storage method for storing the data to be updated in the intermediate data table based on the file generation information of the data already stored in the intermediate data table includes: If it is determined that the file generation information indicates that the existing data has been used to generate the intermediate file, the target storage method for storing the data to be updated in the intermediate data table is determined to be the deletion storage method; If the file generation information indicates that the existing data has not generated the intermediate file, then the storage method for storing the data to be updated in the intermediate data table is determined to be the direct storage method.

4. The method according to claim 1, characterized in that, The process of obtaining the data range of the file to be generated in response to the initiation of a file generation task regarding a source table in the system database includes: In response to the initiation of a file generation task for the source table in the system database, obtain the file generation configuration content related to the file generation task; If the file generation configuration content indicates that a file needs to be generated from the source table in the system database, then the data range of the file to be generated is obtained.

5. The method according to claim 1, characterized in that, The method further includes: Based on the intermediate file identifier information of each of the intermediate files, record the status information of each of the intermediate files in the control table associated with the intermediate files; The process of merging multiple intermediate files to obtain a target file corresponding to the source table includes: If the status information of each of the intermediate files in the control table indicates that the intermediate file was successfully generated, the multiple intermediate files are merged to obtain the target file corresponding to the source table.

6. The method according to claim 1, characterized in that, The system database corresponds to multiple sharded databases, and the data tables obtained after the source table is sharded according to preset rules are stored in the corresponding sharded databases. The step of obtaining target data from multiple intermediate data tables related to the source table according to the data range includes: Based on the data range and the preset rules, generate a data query statement corresponding to the sharded database; According to the data query statement, the target data is obtained from multiple intermediate data tables related to the source table.

7. The method according to claim 1, characterized in that, The generation of intermediate files corresponding to each of the multiple intermediate data tables and the merging of the multiple intermediate files are achieved through streaming reading.

8. A document generation device, characterized in that, The device includes: The first acquisition module is used to acquire the data range of the file to be generated in response to the initiation of a file generation task for the source table in the system database. The second acquisition module is used to acquire target data from multiple intermediate data tables related to the source table according to the data range, wherein the data recorded in each of the multiple intermediate data tables is acquired from a data partition table corresponding to the intermediate table, and the data partition table is obtained after the source table is partitioned; The generation module is used to generate intermediate files corresponding to each of the multiple intermediate data tables based on the target data, thereby obtaining multiple intermediate files; The module is used to merge multiple intermediate files to obtain a target file corresponding to the source table.

9. An electronic device, comprising: One or more processors; Memory, used to store one or more computer programs. The characteristic feature is that the one or more processors execute the one or more computer programs to implement the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program or instructions stored thereon, characterized in that, When the computer program or instructions are executed by a processor, they implement the steps of the method according to any one of claims 1 to 7.

11. A computer program product, comprising a computer program or instructions, characterized in that, When the computer program or instructions are executed by a processor, they implement the steps of the method according to any one of claims 1 to 7.

Images