Data batch transmission method and device

Abstract

The embodiment of the present application provides a method and device for batch data transmission, which can be used in the field of artificial intelligence technology. The method includes: generating a control file corresponding to each batch of host data after batch processing, and the control file is used to record the file information and batch processing time of the corresponding batch of host data; transmitting each batch of host data and the control file to the UDS platform, so that the UDS platform performs data verification on each batch of host data based on the file information and batch processing time in the control file, and performs data splitting and transmission of the verified host data to the downstream of the UDS platform. The present application can effectively improve the timeliness of the UDS platform in processing the received host data, and can effectively improve the integrity and accuracy of the UDS platform in processing the host data, thereby effectively improving the efficiency and automation of batch data transmission to the downstream of the UDS platform.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, in particular to the field of artificial intelligence technology, and specifically to a method and device for batch data transmission. Background Art

[0002] In order to improve efficiency and complete the batch processing of massive data as quickly and timely as possible, the daily batch processing of massive data on the host is generally carried out in parallel. After the host's massive data is processed in parallel batches and a large number of various data files are downloaded, the downstream cross-platform and cross-system UDS platform needs to complete the decoding and splitting processing as soon as possible, and then download or distribute various data files to other open platform systems, channels, and branches for use as soon as possible.

[0003] At present, due to differences in the amount of data processed by each batch, system resources, etc., the time for each batch to complete batch processing and upload data files is different. Therefore, the existing data batch transmission method has the problem that the UDS platform cannot start the host data processing of each batch in time, and cannot guarantee the integrity of the processed data. Summary of the invention

[0004] In response to the problems in the prior art, the present application provides a method and device for batch data transmission, which can effectively improve the timeliness of the UDS platform in processing the received host data, and can effectively improve the integrity and accuracy of the UDS platform in processing the host data, thereby effectively improving the efficiency and automation of batch data transmission to the downstream of the UDS platform.

[0005] In order to solve the above technical problems, this application provides the following technical solutions:

[0006] In a first aspect, the present application provides a method for batch data transmission, comprising:

[0007] Generate a control file corresponding to each batch of host data currently processed in batches, wherein the control file is used to record file information and batch processing time of the corresponding batch of host data;

[0008] The host data of each batch and the control file are transmitted to the UDS platform, so that the UDS platform verifies the host data of each batch based on the file information and batch processing time in the control file, and splits the verified host data and transmits it to the downstream of the UDS platform.

[0009] Furthermore, the generating of the control files corresponding to the host data of each batch currently processed in batches includes:

[0010] According to the host data of each batch, the number, identification, size and batch processing time of the host files corresponding to each batch are obtained respectively;

[0011] Generate a CHK file corresponding to each batch based on the number, identifier and size of the host files corresponding to each batch, wherein the CHK file is used to store the corresponding relationship between the number, identifier and size of the host files;

[0012] A control file corresponding to each batch of host data currently processed in batch is generated according to the CHK file and the batch processing time.

[0013] Furthermore, generating a control file corresponding to each batch of host data currently processed in batches according to the CHK file and the batch processing time includes:

[0014] Generate a CTL file corresponding to each batch based on the batch processing time of the host file corresponding to each batch, wherein the CTL file is used to store the corresponding relationship between the identifier of the host file and the batch processing time;

[0015] The CHK file and the CTL file are used as control files corresponding to each batch of host data currently processed in batches.

[0016] Further, the host data of each batch and the control file are transmitted to the UDS platform, so that the UDS platform verifies the host data of each batch based on the file information and batch processing time in the control file, and splits the verified host data and transmits it to the downstream of the UDS platform, including:

[0017] The host data, CHK files and CTL files of each batch are transmitted to the UDS platform in parallel, so that the UDS platform automatically scans the host data, CHK files and CTL files of each batch, and performs data verification on the host data of each batch based on the CHK files and CTL files, and splits the verified host data and transmits it to the downstream of the UDS platform.

[0018] Furthermore, before generating the control files corresponding to the host data of each batch currently processed in batch, the method further includes:

[0019] Based on a preset division rule, multiple host data are divided into different batches to obtain host data of each batch;

[0020] The host data of each batch is batch processed in parallel.

[0021] Furthermore, the method of dividing the plurality of host data into different batches based on a preset division rule to obtain host data of each batch includes:

[0022] Obtain the correspondence between each pre-stored batch number and the start area code and the end area code;

[0023] According to the area codes corresponding to the host data, and the corresponding relationships between the batch numbers and the start area code and the end area code, the host data are divided into different batches.

[0024] Furthermore, the parallel batch processing of each batch of host data includes:

[0025] Apply pre-set TWS tools to batch process each batch of host data in parallel.

[0026] In a second aspect, the present application provides a data batch transmission device, comprising:

[0027] A file generation module, used to generate a control file corresponding to each batch of host data currently processed in batches, wherein the control file is used to record the file information and batch processing time of the corresponding batch of host data;

[0028] The data transmission module is used to transmit each batch of host data and the control file to the UDS platform, so that the UDS platform can verify the host data of each batch based on the file information in the control file and the batch processing time, and split the verified host data and transmit it to the downstream of the UDS platform.

[0029] In a third aspect, the present application provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the data batch transmission method when executing the program.

[0030] In a fourth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, which implements the data batch transmission method when executed by a processor.

[0031] It can be seen from the above technical scheme that the present application provides a method and device for batch data transmission, the method comprising: generating a control file corresponding to each batch of host data currently processed in batches, wherein the control file is used to record the file information and batch processing time of the corresponding batch of host data; transmitting each batch of host data and the control file to the UDS platform, so that the UDS platform performs data verification on each batch of host data based on the file information and batch processing time in the control file, and performs data splitting on the verified host data and transmits it to the downstream of the UDS platform, by generating a control file corresponding to each batch of host data currently processed in batches, and transmitting each batch of host data and the control file to the downstream of the UDS platform; The system can transfer the files to the UDS platform, realize the automatic and intelligent control of the upstream and downstream file interaction between the host and the open platform, and after the host's massive data is processed and downloaded in batches, the downstream UDS platform can be intelligently identified and automatically controlled to start decoding and splitting processing immediately, which can effectively improve the timeliness of the UDS platform's processing of the received host data, and can effectively improve the integrity and accuracy of the UDS platform's processing of the host data, thereby effectively improving the efficiency and automation of batch data transmission to the downstream of the UDS platform; at the same time, it is non-destructive to the system's batch operation and download, decoding and splitting, and has no effect on the system's own functions and versions; it has universality and generality, and can be widely promoted and used. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0033] Figure 1 It is a schematic diagram of the relationship between the data batch transmission device in the embodiment of the present application and the UDS platform and the client device respectively.

[0034] Figure 2 This is a first flow chart of the method for batch data transmission in an embodiment of the present application.

[0035] Figure 3 It is a second flow chart of the method for batch data transmission in an embodiment of the present application.

[0036] Figure 4 It is a third flow chart of the method for batch data transmission in the embodiment of the present application.

[0037] Figure 5 This is a fourth flow chart of the method for batch data transmission in an embodiment of the present application.

[0038] Figure 6 This is a fifth flow chart of the method for batch data transmission in the embodiment of the present application.

[0039] Figure 7 This is a sixth flow chart of the method for batch data transmission in the embodiment of the present application.

[0040] Figure 8 This is the seventh flow chart of the method for batch data transmission in the embodiment of the present application.

[0041] Fig. 9 It is a structural diagram of a data batch transmission device in an embodiment of the present application.

[0042] Fig.10 It is a flow chart of a data batch transmission method implemented by an application data batch transmission system provided in an application example of the present application.

[0043] Fig.11 It is a structural diagram of a data batch transmission system provided by an application example of this application.

[0044] Fig.12 It is a structural diagram of a massive data host parallel processing module provided in the application example of this application.

[0045] Fig.13 It is a structural diagram of the parallel downloading of massive data from the host to the platform module provided by the application example of this application.

[0046] Fig.14 It is a structural diagram of the platform automatic scanning intelligent control module provided by the application example of this application.

[0047] Fig.15 It is a structural diagram of the platform decoding, splitting and downlinking module provided in the application example of this application.

[0048] Fig.16 It is a schematic diagram of the structure of an electronic device in an embodiment of the present application. DETAILED DESCRIPTION

[0049] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0050] It should be noted that the data batch transmission method and device disclosed in the present application can be used in the fields of automatic control, artificial intelligence and big data technology, and can also be used in any field other than the fields of automatic control, artificial intelligence and big data technology. The application field of the data batch transmission method and device disclosed in the present application is not limited.

[0051] Due to differences in the amount of data processed by each batch, system resources, etc., the time for each batch to complete batch processing and upload data files is different. Therefore, when the downstream UDS platform starts the decoding and splitting processing of each batch is an issue that needs to be resolved.

[0052] The number of data files in each batch is huge and distributed in different branch directories. It is unrealistic to manually check and confirm them one by one before processing: it is inefficient, prone to errors, and will greatly delay the time for UDS to start processing, affecting the supply of data to its downstream open platforms, channels, branches, etc.

[0053] If automatic machine processing is used, it is necessary to set the time for UDS to start automatic processing: if this time is set early, some batches may not have been processed upstream yet; if it is set late enough, it will greatly affect the timeliness of use of UDS and its downstream platforms and channels, which is also unrealistic. Moreover, with the continuous increase in data volume and changes in system performance, even if this time is "late enough", there will still be problems with upstream batches not being processed yet in the future.

[0054] The problem to be solved by the present invention is how to intelligently control the downstream UDS platform to start decoding and splitting processing immediately after each batch of upstream hosts completes batch processing and file downloading, so as to ensure that the host upstream has completed the processing of this batch of data and the download of files (ensuring the integrity and reliability of the upstream data) and start the downstream UDS platform processing immediately (ensuring the high timeliness of the downstream processing) to achieve accurate and efficient automatic control functions.

[0055] Due to the differences in the amount of data processed by each batch, system resources, etc., the time for each batch to complete batch processing and upload data files is different. Therefore, the existing data batch transmission method has the problem that the UDS platform cannot start the host data processing of each batch in time, and cannot ensure the integrity of the processed data. The embodiment of the present application provides a data batch transmission method, which generates a control file corresponding to each batch of host data after batch processing, wherein the control file is used to record the file information and batch processing time of the corresponding batch of host data; the host data of each batch and the control file are transmitted to the UDS platform, so that the UDS platform verifies the host data of each batch based on the file information and batch processing time in the control file, and splits the verified host data and transmits it to the downstream of the UDS platform By generating control files corresponding to each batch of host data after batch processing, and transmitting each batch of host data and the control files to the UDS platform, automatic control and intelligent control of upstream and downstream file interactions between hosts and open platforms are realized. After the host's massive data is processed and downloaded in batches, the downstream UDS platform is intelligently identified and automatically controlled to immediately start decoding and splitting processing, which can effectively improve the timeliness of the UDS platform in processing the received host data, and can effectively improve the integrity and accuracy of the UDS platform in processing the host data, thereby effectively improving the efficiency and automation of batch data transmission to the downstream of the UDS platform; at the same time, it is non-destructive to the system's batch operation and download, decoding and splitting, and has no effect on the system's own functions and versions; it has universality and generality, and can be widely promoted and used.

[0056] Based on the above content, the present application also provides a data batch transmission device for implementing the data batch transmission method provided in one or more embodiments of the present application. The data batch transmission device can be an independent server or a functional module in the existing server system of the financial institution. Figure 1The data batch transmission device can communicate and connect with the UDS platform and each client device by itself or through a third-party server, etc. The data batch transmission device can receive the data batch transmission start instruction sent by the client device, and generate a control file corresponding to each batch of host data after the current batch processing according to the data batch transmission start instruction, wherein the control file is used to record the file information and batch processing time of the corresponding batch of host data; transmit each batch of host data and the control file to the UDS platform, so that the UDS platform performs data verification on each batch of host data based on the file information and batch processing time in the control file, and splits the verified host data and transmits it to the downstream of the UDS platform ((other open platform systems, various channels, various branch servers under financial institutions, etc.), and then, the data batch transmission device can display the data batch transmission result data on the preset monitoring platform and / or send the data batch transmission result data to the client device.

[0057] In one or more embodiments of the present application, the UDS platform refers to the "Unified Data Exchange" platform. The Chinese meaning of UDS is "Unified Data Exchange", and the full English name is "Unified Data Swap". Since the UDS platform is a cross-system and cross-platform open platform, it can no longer be controlled by the host TWS tool (the host TWS tool does not support the open platform). So how to let a certain sub-directory in the Haizi quantum directory of the downstream UDS open platform know that all data files under the "intelligent identification" sub-directory have been collected and collected, and the content is accurate and correct, and quickly and accurately start the decoding and splitting of this sub-directory and other processing and deliver the processing results to its downstream (other open platform systems, various channels, and various branches) in a timely manner for use, is the problem to be solved by this application.

[0058] In one or more embodiments of the present application, the TWS tool refers to the IBM mainframe TWS (Tivoli Workload Scheduler) scheduling tool. The TWS tool supports specific job types including file transfer, database, webservice, Java, script (Shell script, Perl or other application-specific languages), etc.

[0059] In one or more embodiments of the present application, a CHK file refers to a file containing verification information such as application information, number of files, file name, file size, and file structure. The Chinese meaning of CHK is "check, verify, and check", and the full English name is "CHECK"; a CTL file refers to a file containing verification information such as batch processing time. The Chinese meaning of CTL is "management, limitation, constraint", and the full English name is "CONTROL".

[0060] In another practical application scenario, the part of the data batch transmission device that performs data batch transmission can be executed in the server as described above, or all operations can be completed in the user terminal device. The selection can be made based on the processing capability of the user terminal device and the limitations of the user's usage scenario. This application does not limit this. If all operations are completed in the user terminal device, the user terminal device can also include a processor for specific processing of data batch transmission.

[0061] It is understandable that the mobile terminal may include any mobile device capable of loading applications, such as a smart phone, a tablet electronic device, a network set-top box, a portable computer, a personal digital assistant (PDA), a vehicle-mounted device, a smart wearable device, etc. Among them, the smart wearable device may include smart glasses, smart watches, smart bracelets, etc.

[0062] The above-mentioned mobile terminal may have a communication module (i.e., a communication unit) that can communicate with a remote server to achieve data transmission with the server. The server may include a server on the task scheduling center side, and other implementation scenarios may also include a server on an intermediate platform, such as a server on a third-party server platform that has a communication link with the task scheduling center server. The server may include a single computer device, or a server cluster consisting of multiple servers, or a server structure of a distributed device.

[0063] The server and the mobile terminal may communicate with each other using any suitable network protocol, including network protocols that have not yet been developed on the date of filing this application. The network protocols may include, for example, TCP / IP, UDP / IP, HTTP, HTTPS, etc. Of course, the network protocols may also include, for example, RPC (Remote Procedure Call Protocol) and REST (Representational State Transfer) protocols used on top of the above protocols.

[0064] The details are described in detail through the following embodiments and application examples.

[0065] In order to solve the problem that due to the differences in the amount of data processed by each batch, system resources, etc., each batch completes batch processing and downloads data files at different times, so that the existing data batch transmission method has the problem that the UDS platform cannot start the host data processing of each batch in time and cannot ensure the integrity of the processed data, the application provides an embodiment of a data batch transmission method, see Figure 2The data batch transmission method executed by the data batch transmission device specifically includes the following contents:

[0066] Step 100: Generate a control file corresponding to each batch of host data currently processed in batches, wherein the control file is used to record file information and batch processing time of the corresponding batch of host data.

[0067] It is understandable that after the massive host data is processed in batches and in parallel, the massive data information after batch processing needs to be quickly delivered to the downstream platforms, channels, and branches through the UDS platform, and the parallel processed data needs to be transferred as soon as possible. Since its downstream is a cross-platform and cross-system UDS open platform, the host TWS tool can no longer control its downstream to decode and split as soon as possible. In order to enable the downstream UDS open platform to continue processing and downloading quickly and accurately in the first time, step 100 needs to generate a control file during the parallel batch processing process.

[0068] Step 200: Transmit each batch of host data and the control file to the UDS platform, so that the UDS platform verifies the host data of each batch based on the file information in the control file and the batch processing time, and splits the verified host data and transmits it to the downstream of the UDS platform.

[0069] Specifically, the added control file is recorded and generated synchronously while processing and generating various data files. When all the data files under this sub-directory are processed, its control file is also generated synchronously, and can be controlled by the host TWS tool and transmitted to the downstream UDS platform. Since the downstream is the UDS platform, it is cross-system and cross-platform, and can no longer be controlled by the host TWS tool (the host TWS tool does not support the open platform). So how to let a certain sub-directory in the Haizi Quantum directory of the downstream UDS open platform know that all the data files under the "intelligent identification" sub-directory have been collected and collected, and the content is accurate and correct, and quickly and accurately start the decoding and splitting of this sub-directory and other processing and deliver the processing results to its downstream (other open platform systems, various channels, and various branches) in a timely manner for use, is the purpose of adding control files.

[0070] From the above description, it can be seen that the data batch transmission method provided in the embodiment of the present application realizes automatic control and intelligent control of upstream and downstream file interactions between hosts and open platforms by generating control files corresponding to each batch of host data after batch processing, and transmitting each batch of host data and the control files to the UDS platform. After the host's massive data is processed and downloaded in parallel in batches, the downstream UDS platform is intelligently identified and automatically controlled to immediately start decoding and splitting processing, which can effectively improve the timeliness of the UDS platform in processing the received host data, and can effectively improve the integrity and accuracy of the UDS platform in processing the host data, thereby effectively improving the efficiency and automation of batch transmission of data to the downstream of the UDS platform; at the same time, it is non-destructive to the system batch operation and download, decoding and splitting, etc., and has no effect on the system's own functions and versions; it has universality and generality, and can be widely promoted and used.

[0071] In order to improve the efficiency and reliability of generating the control files corresponding to the batches of host data currently processed in batches, an embodiment of the data batch transmission method provided in the present application is provided. Figure 3 , step 100 of the data batch transmission method specifically includes the following contents:

[0072] Step 110: According to the host data of each batch, the number, identification, size and batch processing time of the host files corresponding to each batch are obtained respectively.

[0073] Step 120: Generate a CHK file corresponding to each batch based on the number, identifier and size of the host files corresponding to each batch, wherein the CHK file is used to store the corresponding relationship between the number, identifier and size of the host files.

[0074] Step 130: Generate a control file corresponding to each batch of host data currently processed in batches according to the CHK file and the batch processing time.

[0075] Specifically, the CHK file is used to record all the massive paths in this batch, and process the number of downloaded files, file names, file sizes and other information separately.

[0076] From the above description, it can be seen that the data batch transmission method provided in the embodiment of the present application can effectively improve the efficiency and reliability of generating control files corresponding to each batch of host data after batch processing by generating CHK files corresponding to each batch based on the number, identification and size of the host files corresponding to each batch, thereby effectively improving the accuracy and efficiency of the subsequent UDS platform in verifying the host data of each batch based on the file information in the control file and the batch processing time, so as to further improve the integrity and accuracy of the UDS platform in processing the host data, thereby effectively improving the efficiency and automation of batch data transmission to the downstream of the UDS platform.

[0077] In order to further improve the efficiency and reliability of generating the control files corresponding to the batches of host data currently processed in batches, an embodiment of the data batch transmission method provided in the present application is provided. Figure 4 , step 130 of the data batch transmission method specifically includes the following contents:

[0078] Step 131: Generate a CTL file corresponding to each batch based on the batch processing time of the host file corresponding to each batch, wherein the CTL file is used to store the correspondence between the identifier of the host file and the batch processing time.

[0079] Step 132: Use the CHK file and the CTL file as control files corresponding to each batch of host data currently processed in batches.

[0080] Specifically, in order to enable the downstream UDS open platform to continue processing and downloading quickly and accurately in the first place, the present invention adds control files in the parallel batch processing process: (1) CHK file: records the number of files, file names, file sizes and other information processed and downloaded under all the massive paths under this batch. (2) CTL file: registers the date information of batch processing. While generating various types of data files during daily batch processing, the generation of their CHK files and CTL files is synchronously recorded and then downloaded together: in units of sub-paths, all data files and their CHK files and CTL files under this sub-path are downloaded to the downstream UDS platform.

[0081] While the massive data information after batch processing is downloaded in parallel to the massive path of the downstream open platform, its two control files, namely the CHK file and the CTL file, are downloaded separately.

[0082] The two control files added are recorded and generated synchronously while processing and generating various data files. When all data files in this subdirectory are processed, the two control files are also generated synchronously, and are controlled by the host TWS tool and transmitted to the downstream UDS platform together.

[0083] From the above description, it can be seen that the data batch transmission method provided in the embodiment of the present application generates the CTL files corresponding to each batch based on the batch processing time of the host files corresponding to each batch, which can further improve the efficiency and reliability of generating the control files corresponding to each batch of host data after the current batch processing, and thus can effectively improve the accuracy and efficiency of the subsequent UDS platform in verifying the host data of each batch based on the file information and batch processing time in the control file, so as to further improve the integrity and accuracy of the UDS platform in processing the host data, and thus can effectively improve the efficiency and automation of batch transmission of data to the downstream of the UDS platform.

[0084] In order to improve the efficiency and reliability of the UDS platform to automatically scan each batch of host data, CHK files and CTL files, an embodiment of the data batch transmission method provided in this application is provided. Figure 5 , step 200 of the data batch transmission method specifically includes the following contents:

[0085] Step 210: Transmit each batch of host data, CHK files and CTL files to the UDS platform in parallel, so that the UDS platform automatically scans each batch of host data, CHK files and CTL files, and performs data verification on each batch of host data based on the CHK files and CTL files, and splits the verified host data and transmits it to the downstream of the UDS platform.

[0086] Specifically, the background program of the downstream UDS platform automatically scans and detects the status of file downloads under massive paths. Based on the content information of the control file, it checks whether the batch date, number of files, file name, file size and other information have matched the content information of all the physical data files received under its path. It can intelligently identify which path has completed all the physical data files downloaded, and automatically control it to immediately start the next step of decoding, splitting and downloading.

[0087] Both CHK files and CTL files are used. The detailed application methods are as follows:

[0088] The UDS platform automatically scans and identifies the two control files for verification. When the number of data files under a sub-directory, the name and size of each data file, the file structure and other information are consistent with the information registered in the CHK file under this sub-directory, it is determined that the data files under this sub-directory are collected and complete, and the content is accurate and correct. The "intelligent control is fast and accurate in the first time" is immediately started to decode, split and process all the data files under this sub-directory, and then enter the next module.

[0089] CTL files must also be used. The host parallel batch is repeated every day, and massive data files are generated and downloaded every day. In order to distinguish data files of different dates, the UDS platform contains a date in the file name of each related subdirectory. This date must match the batch date registered in the CTL file before entering the next module.

[0090] From the above description, it can be seen that the data batch transmission method provided in the embodiment of the present application can effectively improve the efficiency and reliability of the UDS platform in automatically scanning each batch of host data, CHK files and CTL files by transmitting each batch of host data, CHK files and CTL files in parallel to the UDS platform, thereby effectively improving the accuracy and efficiency of the subsequent UDS platform in verifying each batch of host data based on the file information and batch processing time in the control file, so as to further improve the integrity and accuracy of the UDS platform in processing host data, thereby effectively improving the efficiency and automation of batch data transmission to the downstream of the UDS platform.

[0091] In order to improve the efficiency and reliability of host data batch processing, an embodiment of the data batch transmission method provided in this application is provided. Figure 6 The method for batch data transmission further includes the following contents before step 100:

[0092] Step 010: Based on a preset division rule, multiple host data are divided into different batches to obtain host data of each batch.

[0093] Step 020: Batch process the host data of each batch in parallel.

[0094] From the above description, it can be seen that the data batch transmission method provided in the embodiment of the present application, by first dividing multiple host data into different batches to obtain host data of each batch, and then batch processing the host data of each batch in parallel, can effectively improve the efficiency and reliability of batch processing of host data, and thus can effectively improve the accuracy and efficiency of the subsequent generation of control files corresponding to the current batches of host data after batch processing, so as to further improve the integrity and accuracy of the UDS platform in processing host data, and thus can effectively improve the efficiency and automation of batch transmission of data to the downstream of the UDS platform.

[0095] In order to improve the efficiency and reliability of dividing multiple host data into different batches, an embodiment of a data batch transmission method provided in the present application is provided. Figure 7 , step 010 in the data batch transmission method specifically includes the following contents:

[0096] Step 011: Obtain the correspondence between each pre-stored batch number and the start area code and the end area code.

[0097] Step 012: According to the area code corresponding to each of the host data, and the correspondence between each batch number and the start area code and the end area code, the host data are divided into different batches.

[0098] Specifically, for the daily processing of massive data on the host, in order to improve processing efficiency and complete the batch processing of massive data as quickly and timely as possible, parallel batch processing is adopted. For example, according to the regional code segments and data volumes of each province and city (municipalities directly under the Central Government and cities with independent planning status), the massive data is divided into multiple batches (numbers) with similar data volumes for daily parallel batch processing.

[0099] If the area code of a province or city belongs to the "starting area code" and "ending area code" segments of a batch (number), then its data belongs to that batch (number). Among them, batch (number) 0 is the whole batch, which contains another 20 batches (numbers), and its "starting area code" and "ending area code" are 0-9999. After the host's massive data is divided into different batches (numbers), its host batch is processed in parallel on a daily basis.

[0100] From the above description, it can be seen that the data batch transmission method provided in the embodiment of the present application can effectively improve the efficiency and reliability of dividing multiple host data into different batches according to the area code corresponding to each of the host data, and the correspondence between each batch number and the starting area code and the ending area code, thereby effectively improving the accuracy and efficiency of subsequent parallel batch processing of host data in each batch, so as to further improve the integrity and accuracy of the UDS platform in processing host data, thereby effectively improving the efficiency and automation of batch data transmission to the downstream of the UDS platform.

[0101] In order to improve the reliability and efficiency of parallel batch processing of host data, an embodiment of a data batch transmission method provided in this application is provided. Figure 8 , step 020 in the data batch transmission method specifically includes the following contents:

[0102] Step 021: Apply the preset TWS tool to batch process each batch of host data in parallel.

[0103] Specifically, step 021 is controlled by the TWS scheduling tool of the IBM mainframe. It can be started automatically or manually, and continue processing after the doubtful links and key links are manually confirmed and responded to. According to the various PLANs set and the various configured parameters, the daily batch parallel processing of the mainframe can be completed quickly and efficiently.

[0104] From the above description, it can be seen that the data batch transmission method provided in the embodiment of the present application can effectively improve the reliability and efficiency of parallel batch processing of host data by applying a preset TWS tool to batch process each batch of host data in parallel, thereby improving the integrity and accuracy of the UDS platform in processing host data, and thereby effectively improving the efficiency and automation of batch transmission of data to the downstream of the UDS platform.

[0105] From the software level, in order to solve the problem that due to the differences in the amount of data processed by each batch, system resources, etc., each batch completes batch processing and downloads data files at different times, so that the existing data batch transmission method has the problem that the UDS platform cannot start the host data processing of each batch in time and cannot ensure the integrity of the processed data, the application provides an embodiment of a data batch transmission device for executing all or part of the content of the data batch transmission method, see Fig. 9 , the data batch transmission device specifically includes the following contents:

[0106] The file generation module 10 is used to generate control files corresponding to each batch of host data currently processed in batches, wherein the control files are used to record file information and batch processing time of the corresponding batch of host data.

[0107] The data transmission module 20 is used to transmit each batch of host data and the control file to the UDS platform, so that the UDS platform can verify the host data of each batch based on the file information and batch processing time in the control file, and split the verified host data and transmit it to the downstream of the UDS platform.

[0108] From the above description, it can be seen that the data batch transmission device provided in the embodiment of the present application realizes automatic control and intelligent control of upstream and downstream file interactions across hosts and open platforms by generating control files corresponding to each batch of host data after batch processing, and transmitting each batch of host data and the control files to the UDS platform. After the host's massive data is processed and downloaded in parallel in batches, the downstream UDS platform is intelligently identified and automatically controlled to immediately start decoding and splitting processing, which can effectively improve the timeliness of the UDS platform in processing the received host data, and can effectively improve the integrity and accuracy of the UDS platform in processing the host data, thereby effectively improving the efficiency and automation of batch transmission of data to the downstream of the UDS platform; at the same time, it is non-destructive to the system batch operation and download, decoding and splitting, etc., and has no effect on the system's own functions and versions; it has universality and generality, and can be widely promoted and used.

[0109] In order to further illustrate this solution, the application example of this application provides a data batch transmission method implemented by an application data batch transmission system. The execution process of the data batch transmission method is shown in Fig.10 It involves the fields of automatic control and artificial intelligence, and realizes automatic and intelligent control of upstream and downstream file interactions across hosts and open platforms. After the host's massive data is processed and transmitted in batches in parallel, the intelligent identification and automatic control downstream UDS platforms immediately start decoding and splitting processing to ensure the accuracy, completeness and reliability of the data obtained downstream, while ensuring the high timeliness of downstream processing.

[0110] See also Fig.11 The data batch transmission system provided by the application example of this application specifically includes four modules, namely, a massive data host parallel processing module 1, a host massive data parallel downlink to the platform module 2, a platform automatic scanning intelligent control module 3 and a platform decoding, splitting and downlinking module 4.

[0111] The specific functions of each module are described as follows:

[0112] 1. Massive data host parallel processing module

[0113] See also Fig.12 The massive data host parallel processing module 1 specifically includes a massive host data batching unit 11 and a TWS control parallel processing unit 12 .

[0114] For the daily processing of massive data on the host, parallel batch processing is used to improve processing efficiency and complete the batch processing of massive data as quickly and timely as possible. For example, according to the regional code segments and data volumes of each province and city (municipalities directly under the Central Government and cities with independent planning status), the massive data is divided into multiple batches (numbers) with similar data volumes for daily parallel batch processing.

[0115] The corresponding relationship between each batch number and the starting area code and the ending area code is shown in Table 1:

[0116] Table 1

[0117] Batch number Starting area code Ending area code 0 0 9999 2 102 299 4 300 499 6 500 699 8 700 999 10 1000 1099 12 1100 1199 14 1200 1299 16 1300 1599 18 1600 1799 20 1800 1999 22 2000 2099 24 2100 2499 26 2500 3099 28 3100 3599 30 3600 3899 32 3900 4599 40 5000 5200 97 4799 4799 98 4600 4700 99 0 101

[0118] If the area code of a province or city belongs to the "starting area code" and "ending area code" segments of a batch (number), then its data belongs to that batch (number). Among them, batch (number) 0 is the entire batch, which includes another 20 batches (numbers), and its "starting area code" and "ending area code" are 0-9999. After the massive data of the mainframe is divided into different batches (numbers), the daily parallel processing of its mainframe batch is controlled by the TWS scheduling tool of the IBM mainframe. It can be started by automatic triggering, started by manual triggering, and continued processing after the doubtful links and key links are manually confirmed and responded to. According to the various PLANs set and various parameters configured, the daily batch parallel processing of the mainframe can be completed quickly and efficiently.

[0119] 2. The host's massive data is transmitted to the platform module in parallel

[0120] See also Fig.13 The host massive data parallel downloading to the platform module 2 specifically includes a host massive data parallel downloading unit 21 and a control file parallel downloading unit 22.

[0121] After the massive data on the mainframe is processed in batches and in parallel, the massive data information after batch processing needs to be quickly delivered to downstream platforms, channels, and branches for use. The data after parallel processing needs to be transmitted as soon as possible. This step is also controlled by the IBM mainframe TWS scheduling tool.

[0122] Since its downstream is the cross-platform and cross-system UDS open platform, the host TWS tool can no longer control its downstream to perform decoding and splitting processing as quickly as possible.

[0123] In order to enable the downstream UDS open platform to continue processing and downloading quickly and accurately in the first place, this application example adds control files during parallel batch processing:

[0124] (1) CHK file: records all the massive paths in this batch, and processes the number of files downloaded, file names, file sizes, and other information.

[0125] (2) CTL file: registers the date information of batch processing. When various data files are generated during daily batch processing, their CHK files and CTL files are recorded and generated simultaneously, and then they are downloaded together: in units of sub-paths, all data files and their CHK files and CTL files under this sub-path are downloaded to the downstream UDS open platform.

[0126] While the massive data information after batch processing is downloaded in parallel to the massive path of the downstream open platform, its two control files, namely the CHK file and the CTL file, are downloaded separately.

[0127] As mentioned above, the two control files added are recorded and generated synchronously while processing and generating various data files. When all data files in this subdirectory are processed, the two control files are also generated synchronously, and are controlled by the host TWS tool and transmitted to the downstream UDS open platform.

[0128] Since the downstream is the UDS open platform, which is cross-system and cross-platform, it can no longer be controlled by the host TWS tool (the host TWS tool does not support the open platform). So how to let a certain sub-directory in the Haizi quantum directory of the downstream UDS open platform know that all data files under the "intelligent identification" sub-directory have been collected and collected, and the content is accurate and correct, and quickly and accurately start the decoding and splitting of this sub-directory and other processing and deliver the processing results to its downstream (other open platform systems, channels, and branches) in time for use? This is the purpose of adding these two control files.

[0129] 3. Platform automatically scans intelligent control module

[0130] See also Fig.14 The platform automatic scanning intelligent control module 3 specifically includes a platform configuration unit 31 , a platform automatic scanning unit 32 and an intelligent identification and control unit 33 .

[0131] The downstream UDS open platform background program automatically scans and detects the file download status under the massive path. Based on the content information of the control file, it checks whether its batch date, number of files, file name, file size and other information have matched the content information of all the physical data files received under its path. It can intelligently identify which physical data files under which path have been completely downloaded, and automatically control it to immediately start the next step of decoding, splitting and downloading.

[0132] Both CHK files and CTL files are used. The detailed application methods are as follows:

[0133] The UDS open platform automatically scans and identifies the two control files for verification. When the number of data files under a sub-directory, the name and size of each data file, the file structure and other information are consistent with the information registered in the CHK file under this sub-directory, it is determined that the data files under this sub-directory are collected and complete, and the content is accurate and correct. The "intelligent control is fast and accurate in the first time" is immediately started to decode, split and process all the data files under this sub-directory, and then enter the next module.

[0134] CTL files must also be used. The host parallel batch is repeated every day, and massive data files are generated and downloaded every day. In order to distinguish data files of different dates, the UDS open platform contains a date in the file name of each related subdirectory. This date must match the batch date registered in the CTL file before entering the next module.

[0135] 4. Platform decoding splitting and downloading module 4

[0136] See also Fig.15 The platform decoding, splitting and downloading module 4 specifically includes a platform decoding, splitting unit 41 and a platform downloading unit 42 .

[0137] It can intelligently identify all physical data files that have been received from the host under a certain path, pass all information verification such as batch date, number of files, file name, file size, file structure, etc., and all content information matches the control file. It can immediately start its UDS decoding and splitting processing to ensure that the data loaded after delivery to its downstream (other open platform systems, various channels, and branches for use) has data integrity, reliability, accuracy, and timeliness.

[0138] The data decoded and split through the UDS open platform is split for downstream use according to the parameters of the configuration unit. For example, the balance sheet files for the big data platform are summarized and provided to the big data platform for use. The huge amount of current account details data is split in parallel to the corresponding distributed open platform database for use, and the data provided to each branch is split by region for use by each branch, etc.

[0139] Specifically, in the application example of this application, while the massive data is processed in parallel batches by the upstream host and the physical massive data files are downloaded, the control files under each massive path are simultaneously recorded and downloaded (the CHK file contains the number of files downloaded from this branch path, the names of each file, the size of each file, and other information, and the CTL contains the host batch date information) for downstream use across platforms and systems. The downstream UDS open platform background program automatically scans and detects the file download status under the massive path, verifies it according to the content information of the control file, and intelligently identifies and determines whether the content information of all physical data files received under a certain path matches its control file information. If the verification is confirmed that all the downloads have been completed, it automatically controls it to immediately start the decoding, splitting, and downloading of its downstream cross-platform and cross-system UDS open platform, and provides the required data files to the downstream open platforms, channels, branches, etc. at the first hand time for use by downstream users and platforms.

[0140] The data batch transmission method implemented by the application data batch transmission system provided in the application example of this application realizes automatic and intelligent control of upstream and downstream file interactions across hosts and open platforms. After the host's massive data is processed and transmitted in batches in parallel, the downstream UDS platform is intelligently identified and automatically controlled to immediately start decoding and splitting processing, ensuring the accuracy, completeness and reliability of the data obtained downstream, while ensuring the high timeliness of downstream processing.

[0141] The advantages of the data batch transmission method implemented by the application data batch transmission system provided in the application example of this application are as follows:

[0142] (1) The added control file unit is non-destructive to the system batch operation, download, decoding and splitting, and has no impact on the system's own functions and versions;

[0143] (2) The intelligent identification and automatic control methods and ideas for massive data interaction across platforms and systems provided by the application examples of this application are universal and general and can be widely promoted and used.

[0144] From the hardware level, in order to solve the problem that due to the differences in the amount of data processed by each batch, system resources, etc., each batch completes batch processing and downloads data files at different times, so that the existing data batch transmission method has the problem that the UDS platform cannot start the host data processing of each batch in time and cannot ensure the integrity of the processed data, the application provides an embodiment of an electronic device for implementing all or part of the contents of the data batch transmission method, and the electronic device specifically includes the following contents:

[0145] Fig.16 FIG. 9 is a schematic block diagram of the system structure of the electronic device 9600 according to an embodiment of the present application. Fig.16 As shown, the electronic device 9600 may include a central processor 9100 and a memory 9140; the memory 9140 is coupled to the central processor 9100. It is worth noting that Fig.16 is exemplary; other types of structures may also be used to supplement or replace this structure to implement telecommunication functions or other functions.

[0146] In one embodiment, the data batch transmission function may be integrated into a central processing unit, wherein the central processing unit may be configured to perform the following control:

[0147] Step 100: Generate a control file corresponding to each batch of host data currently processed in batches, wherein the control file is used to record file information and batch processing time of the corresponding batch of host data.

[0148] It is understandable that after the massive host data is processed in batches and in parallel, the massive data information after batch processing needs to be quickly delivered to the downstream platforms, channels, and branches through the UDS platform, and the parallel processed data needs to be transferred as soon as possible. Since its downstream is a cross-platform and cross-system UDS open platform, the host TWS tool can no longer control its downstream to decode and split as soon as possible. In order to enable the downstream UDS open platform to continue processing and downloading quickly and accurately in the first time, step 100 needs to generate a control file during the parallel batch processing process.

[0149] Step 200: Transmit each batch of host data and the control file to the UDS platform, so that the UDS platform verifies the host data of each batch based on the file information in the control file and the batch processing time, and splits the verified host data and transmits it to the downstream of the UDS platform.

[0150] Specifically, the added control file is recorded and generated synchronously while processing and generating various data files. When all the data files under this sub-directory are processed, its control file is also generated synchronously, and can be controlled by the host TWS tool and transmitted to the downstream UDS platform. Since the downstream is the UDS platform, it is cross-system and cross-platform, and can no longer be controlled by the host TWS tool (the host TWS tool does not support the open platform). So how to let a certain sub-directory in the Haizi Quantum directory of the downstream UDS open platform know that all the data files under the "intelligent identification" sub-directory have been collected and collected, and the content is accurate and correct, and quickly and accurately start the decoding and splitting of this sub-directory and other processing and deliver the processing results to its downstream (other open platform systems, various channels, and various branches) in a timely manner for use, is the purpose of adding control files.

[0151] From the above description, it can be seen that the electronic device provided in the embodiment of the present application realizes automatic control and intelligent control of upstream and downstream file interactions across hosts and open platforms by generating control files corresponding to each batch of host data after batch processing, and transmitting each batch of host data and the control files to the UDS platform. After the host's massive data is processed in parallel and downloaded in batches, the downstream UDS platform is intelligently identified and automatically controlled to immediately start decoding and splitting processing, which can effectively improve the timeliness of the UDS platform in processing the received host data, and can effectively improve the integrity and accuracy of the UDS platform in processing the host data, thereby effectively improving the efficiency and automation of batch transmission of data to the downstream of the UDS platform; at the same time, it is non-destructive to the system's batch operation and download, decoding and splitting, etc., and has no effect on the system's own functions and versions; it has universality and generality, and can be widely promoted and used.

[0152] In another embodiment, the data batch transmission device can be configured separately from the central processing unit 9100. For example, the data batch transmission device can be configured as a chip connected to the central processing unit 9100, and the data batch transmission function can be realized under the control of the central processing unit.

[0153] like Fig.16 As shown, the electronic device 9600 may also include: a communication module 9110, an input unit 9120, an audio processor 9130, a display 9160, and a power supply 9170. It is worth noting that the electronic device 9600 does not necessarily have to include Fig.16 In addition, the electronic device 9600 may also include Fig.16 For components not shown, reference may be made to the prior art.

[0154] like Fig.16 As shown, the central processing unit 9100 is sometimes also referred to as a controller or an operation control, and may include a microprocessor or other processor device and / or logic device. The central processing unit 9100 receives input and controls the operation of various components of the electronic device 9600.

[0155] The memory 9140 may be, for example, one or more of a cache, a flash memory, a hard drive, a removable medium, a volatile memory, a non-volatile memory or other suitable devices. The above-mentioned information related to the failure may be stored, and a program for executing the relevant information may also be stored. The CPU 9100 may execute the program stored in the memory 9140 to implement information storage or processing, etc.

[0156] The input unit 9120 provides input to the central processing unit 9100. The input unit 9120 is, for example, a key or a touch input device. The power supply 9170 is used to provide power to the electronic device 9600. The display 9160 is used to display display objects such as images and texts. The display may be, for example, an LCD display, but is not limited thereto.

[0157] The memory 9140 may be a solid-state memory, such as a read-only memory (ROM), a random access memory (RAM), a SIM card, etc. It may also be a memory that saves information even when the power is off, can be selectively erased, and is provided with more data, examples of which are sometimes referred to as EPROMs, etc. The memory 9140 may also be some other type of device. The memory 9140 includes a buffer memory 9141 (sometimes referred to as a buffer). The memory 9140 may include an application / function storage unit 9142, which is used to store application programs and function programs or processes for executing the operation of the electronic device 9600 through the central processor 9100.

[0158] The memory 9140 may also include a data storage unit 9143 for storing data, such as contacts, digital data, pictures, sounds, and / or any other data used by the electronic device. The driver storage unit 9144 of the memory 9140 may include various drivers for communication functions of the electronic device and / or for executing other functions of the electronic device (such as messaging applications, address book applications, etc.).

[0159] The communication module 9110 is a transmitter / receiver 9110 that sends and receives signals via an antenna 9111. The communication module (transmitter / receiver) 9110 is coupled to the central processor 9100 to provide input signals and receive output signals, which may be the same as the case of a conventional mobile communication terminal.

[0160] Based on different communication technologies, multiple communication modules 9110 may be provided in the same electronic device, such as a cellular network module, a Bluetooth module and / or a wireless LAN module, etc. The communication module (transmitter / receiver) 9110 is also coupled to a speaker 9131 and a microphone 9132 via an audio processor 9130 to provide an audio output via the speaker 9131 and receive an audio input from the microphone 9132, thereby realizing a common telecommunication function. The audio processor 9130 may include any suitable buffer, decoder, amplifier, etc. In addition, the audio processor 9130 is also coupled to the central processor 9100, so that recording can be performed on the local machine through the microphone 9132, and the sound stored on the local machine can be played through the speaker 9131.

[0161] The embodiments of the present application also provide a computer-readable storage medium capable of implementing all the steps of the data batch transmission method in the above embodiments. The computer-readable storage medium stores a computer program. When the computer program is executed by a processor, all the steps of the data batch transmission method in the above embodiments are implemented by the execution subject being a server or a client. For example, when the processor executes the computer program, the following steps are implemented:

[0162] Step 100: Generate a control file corresponding to each batch of host data currently processed in batches, wherein the control file is used to record file information and batch processing time of the corresponding batch of host data.

[0163] It is understandable that after the massive host data is processed in batches and in parallel, the massive data information after batch processing needs to be quickly delivered to the downstream platforms, channels, and branches through the UDS platform, and the parallel processed data needs to be transferred as soon as possible. Since its downstream is a cross-platform and cross-system UDS open platform, the host TWS tool can no longer control its downstream to decode and split as soon as possible. In order to enable the downstream UDS open platform to continue processing and downloading quickly and accurately in the first time, step 100 needs to generate a control file during the parallel batch processing process.

[0164] Step 200: Transmit each batch of host data and the control file to the UDS platform, so that the UDS platform verifies the host data of each batch based on the file information in the control file and the batch processing time, and splits the verified host data and transmits it to the downstream of the UDS platform.

[0165] Specifically, the added control file is recorded and generated synchronously while processing and generating various data files. When all the data files under this sub-directory are processed, its control file is also generated synchronously, and can be controlled by the host TWS tool and transmitted to the downstream UDS platform. Since the downstream is the UDS platform, it is cross-system and cross-platform, and can no longer be controlled by the host TWS tool (the host TWS tool does not support the open platform). So how to let a certain sub-directory in the Haizi Quantum directory of the downstream UDS open platform know that all the data files under the "intelligent identification" sub-directory have been collected and collected, and the content is accurate and correct, and quickly and accurately start the decoding and splitting of this sub-directory and other processing and deliver the processing results to its downstream (other open platform systems, various channels, and various branches) in a timely manner for use, is the purpose of adding control files.

[0166] From the above description, it can be seen that the computer-readable storage medium provided in the embodiment of the present application realizes automatic control and intelligent control of upstream and downstream file interactions across hosts and open platforms by generating control files corresponding to each batch of host data after batch processing, and transmitting each batch of host data and the control files to the UDS platform. After the host's massive data is processed in parallel and downloaded in batches, the downstream UDS platform is intelligently identified and automatically controlled to immediately start decoding and splitting processing, which can effectively improve the timeliness of the UDS platform in processing the received host data, and can effectively improve the integrity and accuracy of the UDS platform in processing the host data, thereby effectively improving the efficiency and automation of batch transmission of data to the downstream of the UDS platform; at the same time, it is non-destructive to the system batch operation and download, decoding and splitting, etc., and has no effect on the system's own functions and versions; it has universality and generality, and can be widely promoted and used.

[0167] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, devices, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0168] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (apparatus), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0169] These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0170] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0171] The present invention uses specific embodiments to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A method for batch data transmission, characterized in that: include: Generate a control file corresponding to each batch of host data currently processed in batches, wherein the control file is used to record file information and batch processing time of the corresponding batch of host data; Transmitting each batch of host data and the control file to the UDS platform, so that the UDS platform verifies each batch of host data based on the file information in the control file and the batch processing time, and splits the verified host data and transmits it to the downstream of the UDS platform; The generating of the control files corresponding to the host data of each batch currently processed in batches includes: According to the host data of each batch, the number, identification, size and batch processing time of the host files corresponding to each batch are obtained respectively; Generate a CHK file corresponding to each batch based on the number, identifier and size of the host files corresponding to each batch, wherein the CHK file is used to store the corresponding relationship between the number, identifier and size of the host files; A control file corresponding to each batch of host data currently processed in batch is generated according to the CHK file and the batch processing time.

2. The method for batch data transmission according to claim 1, characterized in that: The step of generating a control file corresponding to each batch of host data currently processed in batches according to the CHK file and the batch processing time includes: Generate a CTL file corresponding to each batch based on the batch processing time of the host file corresponding to each batch, wherein the CTL file is used to store the corresponding relationship between the identifier of the host file and the batch processing time; The CHK file and the CTL file are used as control files corresponding to each batch of host data currently processed in batches.

3. The method for batch data transmission according to claim 2, characterized in that: The transmitting of each batch of host data and the control file to the UDS platform, so that the UDS platform verifies each batch of host data based on the file information and batch processing time in the control file, and splits and transmits the verified host data to the downstream of the UDS platform, includes: The host data, CHK files and CTL files of each batch are transmitted to the UDS platform in parallel, so that the UDS platform automatically scans the host data, CHK files and CTL files of each batch, and performs data verification on the host data of each batch based on the CHK files and CTL files, and splits the verified host data and transmits it to the downstream of the UDS platform.

4. The method for batch data transmission according to claim 1, characterized in that: Before generating the control files corresponding to the host data of each batch currently processed in batch, the method further includes: Based on a preset division rule, multiple host data are divided into different batches to obtain host data of each batch; The host data of each batch is batch processed in parallel.

5. The method for batch data transmission according to claim 4, characterized in that: The method of dividing the plurality of host data into different batches based on a preset division rule to obtain host data of each batch includes: Obtain the correspondence between each pre-stored batch number and the start area code and the end area code; According to the area codes corresponding to the host data, and the corresponding relationships between the batch numbers and the start area code and the end area code, the host data are divided into different batches.

6. The method for batch data transmission according to claim 4, characterized in that: The batch processing of each batch of host data in parallel includes: Apply pre-set TWS tools to batch process each batch of host data in parallel.

7. A data batch transmission device, characterized in that: include: A file generation module is used to generate a control file corresponding to each batch of host data currently processed in batches, wherein the control file is used to record file information and batch processing time of the corresponding batch of host data, and the generation of the control file corresponding to each batch of host data currently processed in batches includes: According to the host data of each batch, the number, identification, size and batch processing time of the host files corresponding to each batch are obtained respectively; Generate a CHK file corresponding to each batch based on the number, identifier and size of the host files corresponding to each batch, wherein the CHK file is used to store the corresponding relationship between the number, identifier and size of the host files; Generate a control file corresponding to each batch of host data currently processed in batches according to the CHK file and the batch processing time; The data transmission module is used to transmit each batch of host data and the control file to the UDS platform, so that the UDS platform can verify the host data of each batch based on the file information in the control file and the batch processing time, and split the verified host data and transmit it to the downstream of the UDS platform.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the data batch transmission method according to any one of claims 1 to 6 is implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the data batch transmission method described in any one of claims 1 to 6 is implemented.

10. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instruction is executed by a processor, the steps of the data batch transmission method described in any one of claims 1 to 6 are implemented.

Images