A distributed file batch processing method and platform

Abstract

The embodiments of the present specification relate to the field of big data technology, and specifically disclose a distributed file batch processing method and platform, the method comprising: after receiving a batch file transfer request, sending a batch file generation instruction to a batch file generation module deployed in a system node specified by the batch file transfer request, so that the batch file generation module generates a batch file, and feeds back the generated batch file and the file information of the batch file to the data exchange platform; the file information at least includes a file identifier and a file size; after receiving a message of successful batch file generation fed back by each of the specified system nodes, generating a verification file of the batch file transfer request based on the file information corresponding to each of the specified system nodes; and transmitting the batch file to a downstream system based on the verification file, thereby reducing the consumption of system resources while improving the accuracy and integrity of batch file transmission.

Description

Technical Field

[0001] The present specification relates to the field of big data technology, and in particular, to a distributed file batch processing method and platform. Background Art

[0002] With the rapid development of modern information technology, large-scale enterprise systems are gradually moving towards an application system with a clear division of labor, which has resulted in a data exchange platform that meets the needs of batch file exchange between multiple application systems of an enterprise. At the same time, with the rapid growth of the scale of enterprise data, traditional large-scale mainframe databases in the financial industry have transformed into distributed databases. In this process, the complete batch files that can be generated by a single database of the original large-scale mainframe will be composed of batch files corresponding to multiple databases after being separated and stored in distributed databases. In addition, due to the dynamic quantity adjustment requirements of distributed databases, the number and names of batch files generated by each database cannot be fixed, resulting in the inability of the downstream systems that use the files to adapt well to the changes in the large-scale mainframe system.

[0003] There are currently two common solutions. One solution is for the transformed distributed system to perform data merging after generating batch files corresponding to multiple data to avoid affecting the peripheral downstream systems. However, this solution requires the distributed system to design a dedicated data merging device, because the large data scale increases resource consumption, especially when such systems are core business transaction systems, which affects the stability of its external services. Moreover, in the process of enterprises transforming a large number of systems from centralized to distributed, the above solution will obviously increase the overall implementation cost and affect the progress of the enterprise's transformation to distributed. Another solution is for the data exchange platform to merge the files generated by the distributed system, but this solution, on the one hand, is prone to cause cost bottlenecks for the data exchange platform in the scenario of massive file exchange, and increases the time consumption of data exchange, which also has a great impact on the processing timeliness of the data exchange platform. Therefore, there is an urgent need for a more efficient and accurate distributed batch file transmission method. Summary of the invention

[0004] The purpose of the embodiments of this specification is to provide a distributed file batch processing method and platform, which can systematically and low-costly realize data exchange in the process of transformation from an enterprise mainframe to a distributed system, and can improve the accuracy and integrity of file batch transmission.

[0005] This specification provides a distributed file batch processing method and platform which is implemented in the following ways:

[0006] A distributed file batch processing method is applied to a data exchange platform, the method comprising: after receiving a batch file transfer request, sending a batch file generation instruction to a batch file generation module deployed in a system node specified by the batch file transfer request, so that the batch file generation module generates batch files corresponding to the corresponding system node, and feeds back the generated batch files and file information of the batch files to the data exchange platform; the file information at least includes a file identifier and a file size; after receiving a batch file generation success message fed back by each of the specified system nodes, generating a verification file of the batch file transfer request based on the file information corresponding to each of the specified system nodes; and transmitting the batch files to a downstream system based on the verification file.

[0007] In some other embodiments, the file identifier includes at least the file name, the system node identifier and the file generation time.

[0008] In some other embodiments, the verification file and the batch file use different file types.

[0009] In some other embodiments, the verification file is also configured with a first association relationship between a file identifier and a designated downstream system to which the batch files corresponding to the corresponding file identifier are to be transmitted; the transmitting of the batch files to the downstream system based on the verification file includes: for any of the designated downstream systems, extracting all file identifiers corresponding to the designated downstream system from the verification file based on the first association relationship to obtain a file identifier set; and transmitting the batch files corresponding to each file identifier in the file identifier set to the designated downstream system.

[0010] In some other embodiments, the transferring of the batch files corresponding to each file identifier in the file identifier set to the designated downstream system includes: extracting the total file size of the batch files corresponding to each file identifier in the file identifier set from the verification file; and verifying whether the batch files corresponding to each file identifier in the file identifier set have all been transferred to the designated downstream system based on each file identifier and the total file size in the file identifier set.

[0011] In some other embodiments, the verification file also includes a second association relationship between the file identifier and a database table of a designated downstream system into which the batch files corresponding to the corresponding file identifier are to be loaded; and transmitting the batch files to the downstream system based on the verification file includes: when it is determined that all batch files have been transmitted to the designated downstream system, based on the second association relationship, loading the batch files corresponding to each file identifier in the file identifier set into the corresponding database table.

[0012] In some other embodiments, the loading of the batch files corresponding to each file identifier in the file identifier set into the corresponding database table includes: taking each file identifier in the file identifier set as a designated file identifier in turn, searching from a verification file for database table configuration information into which the batch files corresponding to the designated file identifier are to be loaded; and if the database table configuration information cannot be found, issuing an exception reminder that the table information of the designated file identifier is not defined.

[0013] In some other embodiments, the method further includes: processing the batch files based on batch file processing logic; extracting the file size of the processed batch files, and using the extracted file size to update the file size of the corresponding batch files in the verification file to obtain an updated verification file, so as to transmit the processed batch files to a downstream system based on the updated verification file.

[0014] On the other hand, the embodiments of the present specification also provide a data exchange platform, which at least includes a distributed batch device and a file loading device; the distributed batch device at least includes a batch scheduling module, a batch file generation module and a verification file generation module, wherein the batch file generation module is arranged in each system node of the distributed system; the batch scheduling module is used to send a batch file generation instruction to the batch file generation module in the system node specified by the batch file transfer request after receiving the batch file transfer request; the batch file generation module is used to generate a batch file based on the batch file generation instruction, and feed back the generated batch file and the file information of the batch file to the batch scheduling module; the file information at least includes a file identifier and file size; the batch scheduling module is used to send a verification file generation instruction to the verification file generation module after receiving a message of successful batch file generation feedback from each of the designated system nodes; the verification file generation instruction includes file information of the batch files corresponding to each of the designated system nodes; the verification file generation module is used to receive the verification file generation instruction, generate a verification file for the batch file transfer request based on the file information in the verification file generation instruction; and feed back the verification file to the batch scheduling module; the batch scheduling module is used to send the verification file and the batch file to the file loading device; the file loading device is used to transmit the batch file to the downstream system based on the verification file.

[0015] In some other embodiments, the platform also includes a file exchange device; the batch scheduling module is used to send the verification file and the batch file to the file exchange device; the file exchange device is used to process the batch files based on batch file processing logic; and extract the file size of the processed batch file, and use the extracted file size to update the file size of the corresponding batch file in the verification file to obtain an updated verification file; and send the processed batch file and the updated verification file to the file loading device; the file loading device is used to transmit the processed batch file to the downstream system based on the updated verification file.

[0016] On the other hand, an embodiment of the present specification also provides a data exchange platform, which includes at least one processor and a memory for storing processor executable instructions, and when the instructions are executed by the processor, the steps of the method described in any one or more of the above embodiments are implemented.

[0017] The distributed file batch processing method and platform provided in one or more embodiments of this specification can solve the problem of the number and names of batch files not being fixed during the transformation of enterprise mainframes to distributed systems by redesigning the system architecture. After the system is deployed, the distributed system and the data exchange platform can complete the transmission of batch files without merging files, thus avoiding excessive resource consumption caused by file merging during the batch file transmission process. In addition, the downstream system can be compatible with the problem of the number and names of files not being fixed in the upstream system, further ensuring the accuracy and completeness of loading batch files into the downstream system. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the embodiments of this specification or the prior art, the following briefly introduces the drawings required for use in the embodiments or the prior art description. Obviously, the drawings described below are only some embodiments recorded in this specification. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative labor. In the drawings:

[0019] Figure 1 A schematic diagram of the module structure of the data exchange platform provided in this manual;

[0020] Figure 2 The module structure and file transfer process diagram of the distributed batch device provided in this manual;

[0021] Figure 3 The module structure of the file exchange device and the file transfer process diagram provided in this manual;

[0022] Figure 4The module structure of the file loading device and the file transfer process diagram provided in this manual;

[0023] Figure 5 This is a schematic diagram of the implementation process of the distributed file batch processing method provided in this manual. DETAILED DESCRIPTION

[0024] In order to enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in one or more embodiments of this specification will be clearly and completely described below in conjunction with the drawings in one or more embodiments of this specification. Obviously, the described embodiments are only part of the embodiments of the specification, not all of the embodiments. Based on one or more embodiments of the specification, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the embodiments of this specification.

[0025] In a scenario example provided in this specification, the distributed file batch processing method can be applied to a data exchange platform. Figure 1 As shown, the data exchange platform may include a distributed batch device 1 , a file exchange device 2 and a file loading device 3 .

[0026] The distributed batch device 1 is connected to the distributed system and is used to process the data of each system node of the distributed system to generate a batch file corresponding to each system node; and record the file information of the generated batch file into a verification file. The distributed batch device 1 can also be connected to the file exchange device 2 to transmit the generated batch file and the verification file to the file exchange device 2.

[0027] The file exchange device 2 is also connected to the file loading device 3. The file exchange device 2 can filter, clean and process the batch files, and transmit the processed batch files and the verification files to the file loading device 3.

[0028] The file loading device 3 is also connected to the database of the peripheral downstream system. The file loading device 3 can verify the batch files according to the file information in the verification file, and after the verification is correct, load the multiple batch files into the database of the peripheral downstream system.

[0029] Figure 2 FIG. 1 is a schematic diagram of the system architecture and file processing flow of the distributed batch device 1. Figure 2As shown, the distributed batch device 1 may at least include a batch scheduling module, a batch file generation module and a verification file generation module. The batch file generation module is arranged in each system node to process the data in the database under the corresponding system node and generate a batch file corresponding to the corresponding system node. The configuration center of the platform may pre-store the database configuration information under each system node of each distributed system. The database configuration information may at least include the system node to which the database belongs.

[0030] Accordingly, the distributed batch device 1 can execute the following steps 201-206 to perform batch processing of files.

[0031] Step 201: After receiving the batch file transfer request, the batch scheduling module may send a request to the configuration center to read the database configuration information of each system node. The batch file transfer request may include the system node to which the data to be transferred in this batch file transfer belongs, the database under the system node, the table information in the database, and the method of performing processing on the data in the database. Alternatively, it may also include the configuration information of the downstream system to which each batch file is to be transferred, the database table configuration information of the downstream system to which the data in the batch file is to be loaded, and the like. The generation of the batch file transfer request and the content included therein may be configured as needed and are not limited here. The configuration center may be pre-configured with the configuration information of each system node in the distributed system, the configuration information of the downstream system, and the like.

[0032] Step 202: After receiving the above request from the batch scheduling module, the configuration center may return the stored database configuration information to the batch scheduling module.

[0033] Step 203: The batch scheduling module sends a batch file generation instruction to the batch file generation module under the system node specified by the batch file transfer request according to the database configuration information of the distributed system.

[0034] Step 204: After receiving the batch file generation instruction, the batch file generation module can process the data in the database under the system node according to the batch file transfer request to generate a batch file. And return the file information of the generated batch file to the batch scheduling module. The file information may at least include a file identifier and a file size. The file identifier may at least include a file name, a system node identifier, and a file generation time. For example, the file identifier may be: [file English name]-[distributed database number]-[date].BIN. By adding the distributed database number as part of the file identifier, it can be ensured that the file identifiers generated by the databases under different system nodes are not repeated, and at the same time, it is also convenient for tracing.

[0035] Step 205: After receiving the instruction that the batch files are generated successfully from each of the designated system nodes, the batch scheduling module sends a start instruction for generating the verification files to the verification file generation module. The verification file generation instruction at least includes the file information received from each system node.

[0036] Step 206: After receiving the verification file generation instruction, the verification file generation module generates a verification file and returns the result to the batch scheduling module. The file identification rule of the verification file can be "[file English name]-[date].CHK". The file identification of the batch file is kept similar, but the suffix is ​​different to distinguish the verification file from the batch file. The content and specific format of the verification file are preferably:

[0037] (1) The first record of the verification file may include the overall information of the batch files required to be transferred by the batch file transfer request. Table 1 is an example table of field information included in the first record.

[0038] Table 1

[0039] Field Name Field Description APP_ID Application ID of the source application APP_NAME Source application name BIZ_DATE File transfer date FILE_COUNT Number of batch files BAK_FIELD Alternative fields

[0040] (2) The second to last records are file information. Table 2 is an example table of field information of the second to last records.

[0041] Table 2

[0042]

[0043]

[0044] Figure 3 The system architecture of the file exchange device 2 and the schematic diagram of the processing flow of batch files. The file exchange device distributes the batch files and verification files received from the distributed batch device 1 to the file loading device associated with the downstream system according to the pre-set configuration. The file exchange device 2 may include a file receiving module, a file processing module, and a file distribution module. The configuration center of the platform may also be configured with a batch file processing method (such as field format conversion, decoding method, filtering configuration) and the target downstream system to which the batch files should be distributed (for example, M files to system A, system B; N files to system A, system C, etc.). Alternatively, the target downstream system to which the batch files should be distributed and the configuration information of the database table to be loaded into the verification file may also be recorded. Accordingly, the verification file may be configured with a first association relationship between a file identifier and a designated downstream system to which the batch files corresponding to the corresponding file identifier are to be transmitted; and a second association relationship between a file identifier and a database table of a designated downstream system to which the batch files corresponding to the corresponding file identifier are to be loaded.

[0045] Accordingly, the file exchange device 2 may execute the following steps 301 to 303 to perform batch processing of files.

[0046] Step 301: The file receiving module may receive the batch files and verification files sent by the distributed batch device 1, and provide them to the file processing module for processing.

[0047] Step 302: The file processing module may retrieve the batch file processing method from the configuration center to perform filtering, cleaning and other processing on the batch files, and update the file size of the processed batch files into the verification file.

[0048] Step 303: The file distribution module may distribute the batch files processed by the file processing module and the updated verification files to the file loading device 3 associated with the corresponding designated downstream system based on the first association relationship.

[0049] Figure 4 The system architecture of the file loading device 3 and the processing flow diagram of batch files are shown. The file loading device cooperates with the peripheral downstream system to receive the batch files and verification files provided by the file exchange device 2, and loads all batch files into the database table of the corresponding downstream system according to the file information in the verification file, so as to simply and effectively adapt the downstream system to the problem of the number and name of files in the distributed system being not fixed. Each file loading device 3 can use steps 401 to 404 to load the batch files into the database table of the downstream system.

[0050] Step 401: Determine whether the batch files corresponding to the verification file have arrived. Specifically, based on the file information in the verification file, through the file identifier "[file English name]-[distributed database number]-[date]", it can be determined one by one whether the batch files corresponding to the downstream system associated with the file loading device have arrived at the downstream system. When it is confirmed that each batch file has arrived, and the total file size of the actual batch file is consistent with that recorded in the verification file (verifying the integrity of the batch file), proceed to step 402; otherwise, continue to wait for a period of time, and re-execute the batch file transmission and step 401.

[0051] Step 402: Based on the second association relationship, search the verification file for the database table configuration information of the downstream system to which each file is to be loaded, wherein the database table configuration information may include the database and table information to be loaded. If the corresponding database and table information is successfully found, execute step 404; otherwise, execute step 403.

[0052] Step 403: If the corresponding database table configuration information is not successfully found in step 402, an exception reminder of "verification file undefined" is thrown.

[0053] Step 404: According to the database table configuration information found in step 402 and the "file identifier" recorded in the verification file, the corresponding file is loaded into the corresponding database table.

[0054] Repeat the above steps until all batch files corresponding to the downstream system associated with the file loading device are loaded. The above system architecture is used to perform batch file transmission, so that the batch files can be transmitted without merging the batch files of each system node, thereby avoiding the problem of excessive resource consumption caused by batch file merging. Even if the number and name of files in the distributed system are not fixed, the batch files can be accurately and completely loaded into the database table of the downstream system, thereby improving the accuracy and completeness of batch file loading. And if there is a problem that cannot be loaded, the source of the problematic file can be quickly traced back to improve the efficiency of correcting abnormal file loading.

[0055] Based on the above scenario example, the embodiment of this specification provides a distributed batch file transmission method, such as Figure 5 shown. Figure 5 The following is a flow chart of an embodiment of a distributed file batch processing method provided in this specification. The method can be applied to a data exchange platform, and the method can include the following steps.

[0056] S52: after receiving the batch file transfer request, sending a batch file generation instruction to a batch file generation module deployed in the system node specified by the batch file transfer request, so that the batch file generation module generates batch files corresponding to the corresponding system node, and feeds back the generated batch files and file information of the batch files to the data exchange platform; the file information at least includes a file identifier and a file size;

[0057] S54: after receiving the message of successful batch file generation fed back by each of the designated system nodes, generating a verification file of the batch file transfer request based on the file information corresponding to each of the designated system nodes;

[0058] S56: Transmitting the batch file to a downstream system based on the verification file.

[0059] By adopting the above method to perform batch file transmission, batch file transmission can be realized without merging batch files of each system node, thus avoiding the problem of excessive resource consumption caused by batch file merging. Even if the number and names of files in the distributed system are not fixed, batch files can be accurately and completely loaded into the database table of the downstream system, thus improving the accuracy and completeness of batch file loading.

[0060] In some other embodiments, the file identifier may include at least a file name, a system node identifier, and a file generation time. The file name may be generated according to the content contained in the batch file, or may be generated by a random code. The system node identifier may be, for example, a database encoding under the system node. By configuring the file identifier in the above manner, it is possible to avoid different file identifiers generated by different system nodes at different time points, facilitate accurate processing of files during batch file transmission, and improve the accuracy of batch file transmission and loading.

[0061] In some other embodiments, the verification file and the batch file use different file types. By using different file types, it is easy to distinguish the verification file and the batch file. For example, different file identification suffixes can be used to distinguish the file types of the verification file and the batch file. For example, the batch file uses "BIN" as the suffix of the file identification, and the verification file uses "CHK" as the suffix of the file identification.

[0062] In some other embodiments, the verification file is further configured with a first association relationship between a file identifier and a designated downstream system to which the batch files corresponding to the corresponding file identifier are to be transmitted. The transmitting the batch files to the downstream system based on the verification file may include: for any of the designated downstream systems, extracting all the file identifiers corresponding to the designated downstream system from the verification file based on the first association relationship to obtain a file identifier set; and transmitting the batch files corresponding to each file identifier in the file identifier set to the designated downstream system.

[0063] In some other embodiments, the transferring of the batch files corresponding to each file identifier in the file identifier set to the designated downstream system may include: extracting the total file size of the batch files corresponding to each file identifier in the file identifier set from the verification file; and verifying whether all the batch files corresponding to each file identifier in the file identifier set have been transferred to the designated downstream system based on each file identifier and the total file size in the file identifier set. By comprehensively considering all the file identifiers and the total file size of the batch files corresponding to the downstream system and verifying whether all the batch files corresponding to each downstream system have arrived, the integrity of the batch file transfer can be determined more accurately. When all the files have arrived, the subsequent file loading step can be executed to further improve the accuracy of file loading.

[0064] In some other embodiments, the verification file further includes a second association relationship between the file identifier and a database table of a designated downstream system to which the batch files corresponding to the corresponding file identifier are to be loaded. The transmitting the batch files to the downstream system based on the verification file may include: when it is determined that all the batch files have been transmitted to the designated downstream system, based on the second association relationship, loading the batch files corresponding to each file identifier in the file identifier set into the corresponding database table. By configuring the above second association relationship in the verification file, the efficiency and accuracy of file loading in each downstream system can be greatly improved.

[0065] In some other embodiments, the step of loading the batch files corresponding to each file identifier in the file identifier set into the corresponding database table includes: taking each file identifier in the file identifier set as a designated file identifier in turn, searching the verification file for the database table configuration information to be loaded into the batch files corresponding to the designated file identifier; and issuing an exception reminder that the table information of the designated file identifier is undefined if the database table configuration information cannot be found. By executing file transfer and loading in the above manner, an exception reminder can be issued based on the file identifier when an exception occurs in file loading, thereby greatly improving the efficiency of file loading exception tracing and correcting the exception in a timely manner.

[0066] In some other embodiments, the method further includes: the file exchange device processes the batch files based on the batch file processing logic; and extracts the file size of the processed batch files, and uses the extracted file size to update the file size of the corresponding batch files in the verification file to obtain an updated verification file, so as to transmit the processed batch files and the updated verification file to the file loading device associated with the corresponding downstream system. The batch file processing logic can be pre-configured. Preferably, the batch file processing logic can be configured separately based on the file identifier. Accordingly, the processing logic identifier corresponding to each processing logic can also be recorded in the verification file. During the batch file transmission process, the processing logic identifier corresponding to each file identifier can be read from the verification file to retrieve the processing logic corresponding to the processing logic identifier, and the batch files corresponding to the corresponding file identifier can be processed to improve the processing efficiency.

[0067] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. For details, please refer to the description of the above-mentioned related processing related embodiments, and no further description is given here.

[0068] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0069] Based on the methods and system architecture provided in the above embodiments, the embodiments of this specification also provide a data exchange platform, which may include at least a distributed batch device and a file loading device; the distributed batch device includes at least a batch scheduling module, a batch file generation module and a verification file generation module, wherein the batch file generation module is deployed in each system node of the distributed system.

[0070] The batch scheduling module may be configured to, after receiving a batch file transfer request, send a batch file generation instruction to a batch file generation module in a system node specified by the batch file transfer request.

[0071] The batch file generation module can be used to generate batch files based on the batch file generation instruction, and feed back the generated batch files and file information of the batch files to the batch scheduling module; the file information at least includes a file identifier and a file size.

[0072] The batch scheduling module can be used to send a verification file generation instruction to the verification file generation module after receiving a message indicating that the batch file generation is successful from each of the designated system nodes; the verification file generation instruction includes file information of the batch files corresponding to each of the designated system nodes.

[0073] The verification file generation module can be used to receive the verification file generation instruction, generate the verification file of the batch file transfer request based on the file information in the verification file generation instruction; and feed the verification file back to the batch scheduling module.

[0074] The batch scheduling module may be used to send the verification file and the batch file to the file loading device.

[0075] The file loading device may be used to transmit the batch file to a downstream system based on the verification file.

[0076] In some other embodiments, the platform may further include a file exchange device. Accordingly, the batch scheduling module may be used to send the verification file and the batch file to the file exchange device. The file exchange device may be used to process the batch file based on the batch file processing logic; and extract the file size of the processed batch file, and use the extracted file size to update the file size of the corresponding batch file in the verification file to obtain an updated verification file; and send the processed batch file and the updated verification file to the file loading device. The file loading device may be used to transmit the processed batch file to the downstream system based on the updated verification file.

[0077] It should be noted that the platform described above may also include other implementations according to the description of the above embodiment. The specific implementation methods may refer to the description of the relevant method embodiments, which will not be described one by one here.

[0078] This specification also provides a data exchange platform, which may include at least one processor and a memory for storing processor executable instructions, and the instructions, when executed by the processor, implement the steps of the method described in any one or more of the above embodiments. The memory may include a physical device for storing information, which is usually to digitize the information and then store it in a medium using electrical, magnetic or optical methods. The storage medium may include: devices that store information using electrical energy, such as various memories, such as RAM, ROM, etc.; devices that store information using magnetic energy, such as hard disks, floppy disks, magnetic tapes, magnetic core memories, bubble memories, and U disks; devices that store information optically, such as CDs or DVDs. Of course, there are other types of readable storage media, such as quantum memories, graphene memories, and so on.

[0079] It should be noted that the embodiments of this specification are not limited to the situations that must comply with the standard data model / template or the embodiments of this specification. Certain industry standards or slightly modified implementation plans based on the implementation described in the custom method or the embodiment can also achieve the same, equivalent or similar, or predictable implementation effects after deformation of the above-mentioned embodiments. The embodiments obtained by using these modified or deformed data acquisition, storage, judgment, processing methods, etc. can still fall within the scope of the optional implementation plans of this specification.

[0080] Each embodiment in this specification is described in a progressive manner, and the same and similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment. In the description of this specification, the description of the reference terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of this specification. In this specification, the schematic representation of the above terms does not necessarily target the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner. In addition, in the absence of contradiction, a person skilled in the art can combine and combine the different embodiments or examples described in this specification and the features of different embodiments or examples.

[0081] The above description is only an embodiment of the present specification and is not intended to limit the present specification. For those skilled in the art, the present specification may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present specification shall be included in the scope of the claims of the present specification.

Claims

1. A distributed file batch processing method, characterized in that: Applied to a data exchange platform, the method comprises: After receiving the batch file transfer request, sending a batch file generation instruction to the batch file generation module deployed in the system node specified by the batch file transfer request, so that the batch file generation module generates batch files corresponding to the corresponding system node, and feeds back the generated batch files and file information of the batch files to the data exchange platform; the file information at least includes a file identifier and a file size; After receiving a message that the batch files are successfully generated, based on the file information corresponding to each of the designated system nodes, generating a verification file for the batch file transfer request; transmitting the batch file to a downstream system based on the verification file; The verification file is also configured with a first association relationship between the file identifier and a designated downstream system to which the batch files corresponding to the corresponding file identifier are to be transmitted; The transmitting the batch files to the downstream system based on the verification file includes: for any of the designated downstream systems, extracting all file identifiers corresponding to the designated downstream system from the verification file based on the first association relationship to obtain a file identifier set; and transmitting the batch files corresponding to each file identifier in the file identifier set to the designated downstream system.

2. The method according to claim 1, characterized in that The file identifier at least includes the file name, the system node identifier and the file generation time.

3. The method according to claim 1, characterized in that The verification file and the batch file are of different file types.

4. The method according to claim 1, characterized in that: The transferring of the batch files corresponding to each file identifier in the file identifier set to the designated downstream system includes: Extracting the total file size of the batch files corresponding to each file identifier in the file identifier set from the verification file; Based on each file identifier in the file identifier set and the total file size, it is verified whether the batch files corresponding to each file identifier in the file identifier set have all been transmitted to the designated downstream system.

5. The method according to claim 4, characterized in that The verification file further includes a second association relationship between the file identifier and a database table of a designated downstream system to which the batch files corresponding to the corresponding file identifier are to be loaded; The transmitting the batch files to the downstream system based on the verification file includes: when it is determined that all the batch files have been transmitted to the designated downstream system, based on the second association relationship, loading the batch files corresponding to each file identifier in the file identifier set into a corresponding database table.

6. The method according to claim 5, characterized in that The step of loading the batch files corresponding to each file identifier in the file identifier set into the corresponding database table includes: Taking each file identifier in the file identifier set as a designated file identifier in turn, searching the verification file for database table configuration information to be loaded into the batch files corresponding to the designated file identifier; In the case where the database table configuration information cannot be found, an abnormal reminder is issued that the table information identified by the specified file is undefined.

7. The method according to claim 1, characterized in that The method further comprises: Processing the batch files based on batch file processing logic; Extract the file size of the processed batch files, use the extracted file size to update the file size of the corresponding batch files in the verification file, and obtain an updated verification file, so as to transmit the processed batch files to the downstream system based on the updated verification file.

8. A data exchange platform, characterized in that: The platform at least includes a distributed batch device and a file loading device; the distributed batch device at least includes a batch scheduling module, a batch file generation module and a verification file generation module, wherein the batch file generation module is arranged in each system node of the distributed system; The batch scheduling module is used for sending a batch file generation instruction to a batch file generation module in a system node specified by the batch file transfer request after receiving the batch file transfer request; The batch file generation module is used to generate batch files based on the batch file generation instruction, and feed back the generated batch files and file information of the batch files to the batch scheduling module; the file information at least includes a file identifier and a file size; The batch scheduling module is used to send a verification file generation instruction to the verification file generation module after receiving a message that the batch file generation is successful from each of the designated system nodes; the verification file generation instruction includes file information of the batch file corresponding to each of the designated system nodes; The verification file generation module is used to receive the verification file generation instruction, generate the verification file of the batch file transfer request based on the file information in the verification file generation instruction; and feed the verification file back to the batch scheduling module; The batch scheduling module is used to send the verification file and the batch file to the file loading device; The file loading device is used to transmit the batch file to the downstream system based on the verification file; The verification file is also configured with a first association relationship between the file identifier and a designated downstream system to which the batch files corresponding to the corresponding file identifier are to be transmitted; The file loading device is specifically used to: for any of the designated downstream systems, extract all file identifiers corresponding to the designated downstream system from the verification file based on the first association relationship to obtain a file identifier set; and transfer batch files corresponding to each file identifier in the file identifier set to the designated downstream system.

9. The platform according to claim 8, characterized in that The platform also includes a file exchange device; The batch scheduling module is used to send the verification file and the batch file to the file exchange device; The file exchange device is used to process the batch files based on the batch file processing logic; and extract the file size of the processed batch files, and use the extracted file size to update the file size of the corresponding batch files in the verification file to obtain an updated verification file; and sending the processed batch files and updated verification files to the file loading device; The file loading device is used to transmit the processed batch files to the downstream system based on the updated verification file.

10. A data exchange platform, characterized in that: The platform comprises at least one processor and a memory for storing processor-executable instructions, wherein the instructions implement the steps of the method according to any one of claims 1 to 7 when executed by the processor.

Images