A file transfer processing method and device for a bulk data bus

Abstract

The application discloses a file transmission processing method and device of a batch data bus. When a file stored in a storage area network is received, a control file is triggered to be generated. The control file is used for marking whether the file is processed or not. All upstream original file directories are scanned. If there is a file in all the upstream original file directories, files meeting preset batch forwarding specifications are selected from all the upstream original file directories. When each forwarding task is acquired, the files meeting the preset batch forwarding specifications are batch-sent to a downstream system through preset batch forwarding rules and a batch data bus file transmission channel. When the files meeting the preset batch forwarding specifications are all sent to the downstream system, the files batch-forwarded to the downstream system are marked through the control file.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of file transmission processing, and more particularly to a file transmission processing method and device of a batch data bus. BACKGROUND

[0002] With the increase of the demand for file transmission using a batch data bus in the process of moving the related distributed core downward, the developers of the related project team need to configure more and more transmission files and tasks, but most of the files to be transmitted will be subscribed by different downstream systems, and the subscription is a very homogeneous task. Since the manual configuration of the file processing task is time-consuming and affects the efficiency, it will reduce the processing efficiency of the file transmission in the batch data bus.

[0003] Therefore, how to improve the efficiency of file transmission in the batch data bus is a problem to be solved by the present application. SUMMARY

[0004] Therefore, the present application discloses a file transmission processing method and device of a batch data bus, which aims to reduce the redundancy of repeated sending of files and improve the efficiency of file transmission in the batch data bus.

[0005] In order to achieve the above-mentioned purpose, the disclosed technical solution is as follows:

[0006] The first aspect of the present application discloses a file transmission processing method of a batch data bus, which comprises:

[0007] When receiving a file stored in a storage area network, a control file is triggered to be generated; the control file is used to mark whether the file is processed or not;

[0008] Scanning all upstream original file directories, if there are files in all upstream original file directories, selecting files meeting a preset batch forwarding specification from all upstream original file directories;

[0009] When obtaining each forwarding task, the files meeting the preset batch forwarding specification are batched and sent to a downstream system through a preset batch forwarding rule and a batch data bus file transmission channel;

[0010] When the files meeting the preset batch forwarding specification are all sent to the downstream system, the files batched and forwarded to the downstream system are marked through the control file.

[0011] Preferably, when the file stored in the storage area network is received, the control file is triggered to be generated, which comprises:

[0012] When receiving the file uploaded by the upstream application system to the storage area network storage locally of the server, a control file with the same name as the file stored in the storage area network is generated.

[0013] Preferably, the scanning of all upstream original file directories is performed, and if there are files in all the upstream original file directories, files meeting the preset batch forwarding specification are selected from all the upstream original file directories, including:

[0014] All upstream original file directories are scanned by a monitoring script;

[0015] If there are files in all the upstream original file directories, a file processing thread is notified to work;

[0016] The file processing thread selects files consistent with the fixed file format issued by the batch data bus system node from all the upstream original file directories, and determines the files consistent with the fixed file format issued by the batch data bus system node as the files meeting the preset batch forwarding specification.

[0017] Preferably, when each forwarding task is acquired, the files meeting the preset batch forwarding specification are batch sent to the downstream system through the preset batch forwarding rule and the batch data bus file transmission channel, including:

[0018] When each forwarding task is acquired, the original file meeting the preset specification is sent to a temporary directory through a cut command;

[0019] The file in the temporary directory is copied to a backup file directory through a copy command to perform file backup;

[0020] After the file backup is completed, the files in the backup directory are respectively copied to different forwarding downstream directories, and the completion time of each downstream directory is recorded;

[0021] For the completion time of each downstream directory, the last time from the temporary directory to the target downstream directory is moved within the completion time until there is no source file in the temporary folder in the temporary directory, so as to complete the process of moving the source file directory to the file in the to-be-forwarded directory;

[0022] According to the forwarding task batch, the file in the to-be-forwarded directory is forwarded to the respective corresponding downstream directory, so as to complete the process of sending the file meeting the preset batch forwarding specification to the downstream system.

[0023] Preferably, it further includes:

[0024] In the process of forwarding the files in the to-be-forwarded directory to the respective corresponding downstream directory according to the forwarding task batch, if the machine is monitored to be down, the machine is restarted, the relevant files after the success are sent to the temporary directory, and the step of copying the files in the temporary directory to the backup directory through a copy command to perform file backup is performed.

[0025] Preferably, after the step of copying the files in the temporary directory to the backup directory through a copy command to perform file backup, the method further comprises:

[0026] Recording the time of backup and the directory of backup.

[0027] Preferably, after the process of forwarding the files in the to-be-forwarded directory to the respective corresponding downstream directory according to the forwarding task batch to complete the sending of the files meeting the preset batch forwarding specification to the downstream system, the method further comprises:

[0028] Generating an encrypted file in the temporary directory through an encryption algorithm, and marking the temporary folder through the encrypted file.

[0029] The second aspect of the present application discloses a file transmission processing device of a batch data bus, the device comprising:

[0030] A triggering unit is configured to trigger the generation of a control file when receiving a file stored in a storage area network; the control file is used to mark whether the file is processed or not;

[0031] A scanning and selecting unit is configured to scan all upstream original file directories, and select files meeting a preset batch forwarding specification from all the upstream original file directories if there are files in all the upstream original file directories;

[0032] A batch sending unit is configured to batch send the files meeting the preset batch forwarding specification to a downstream system through a preset batch forwarding rule and a batch data bus file transmission channel when obtaining each forwarding task;

[0033] A marking unit is configured to mark the files batch forwarded to the downstream system through the control file when the files meeting the preset batch forwarding specification are all sent to the downstream system.

[0034] Preferably, the triggering unit is specifically configured to:

[0035] Trigger the generation of a control file with the same name as a file stored in a storage area network uploaded to a server local by an upstream application system when receiving the file.

[0036] Preferably, the scanning and selecting unit comprises:

[0037] The scanning module is configured to scan all upstream original file directories by monitoring scripts;

[0038] The notification module is configured to notify the file processing thread to work if there are files in all upstream original file directories.

[0039] The selection determination module is configured to select, by the file processing thread, files consistent with the fixed file format issued by the batch data bus from all upstream original file directories, and determine the files consistent with the fixed file format issued by the batch data bus system node as files meeting the preset batch forwarding specification.

[0040] According to the technical solution, when a file stored in a storage area network is received, a control file is generated, the control file is used to mark whether the file is processed, all upstream original file directories are scanned, if there are files in all upstream original file directories, files meeting the preset batch forwarding specification are selected from all upstream original file directories, when each forwarding task is acquired, the files meeting the preset batch forwarding specification are batch sent to a downstream system through a preset batch forwarding rule and a batch data bus file transmission channel, and when the files meeting the preset batch forwarding specification are all sent to the downstream system, the files batch forwarded to the downstream system are marked through the control file.

[0041] According to the above solution, when the configurable requirement of each forwarding task is acquired, the data of the upstream system is sent to the downstream system through the batch data bus file transmission channel according to the configurable requirement, the file transmission between the end-to-end can also be realized according to the configurable file forwarding requirement, each requirement only needs to configure forwarding once, so as to solve the file transmission requirement from the end-to-end. For the file stored in the batch data bus system, the files meeting the preset batch forwarding specification are batch sent to the downstream system according to the preset batch forwarding rule and the batch data bus file transmission channel, the files are copied, moved and backed up according to the forwarding requirement table of the file subscription in the process of batch sending the files meeting the preset batch forwarding specification to the downstream system, so as to ensure the continuity and reliability of the file operation, reduce the redundancy of repeated file sending, and improve the efficiency of the file transmission in the batch data bus. BRIEF DESCRIPTION OF DRAWINGS

[0042] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed in the embodiments or the prior art description will be briefly introduced. Obviously, the drawings in the following description are only embodiments of the present application, and those skilled in the art can obtain other drawings according to the provided drawings without creating any creative labor.

[0043] Figure 1 A flowchart of a file transmission processing method of a batch data bus disclosed in an embodiment of the present application is shown in FIG. 1.

[0044] Figure 2 A schematic diagram of a file processing task disclosed in an embodiment of the present application is shown in FIG. 2.

[0045] Figure 3 A schematic diagram of a file processing judgment flow disclosed in an embodiment of the present application is shown in FIG. 3.

[0046] Figure 4 An architecture diagram of a file transmission processing of a batch data bus disclosed in an embodiment of the present application is shown in FIG. 4.

[0047] Figure 5 A structural schematic diagram of a file transmission processing device of a batch data bus disclosed in an embodiment of the present application is shown in FIG. 5.

[0048] Figure 6 A structural schematic diagram of an electronic device disclosed in an embodiment of the present application is shown in FIG. 6. DETAILED DESCRIPTION

[0049] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work fall within the scope of protection of the present application.

[0050] In the present application, the terms “comprise”, “contain” or any other variants thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device comprising a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such process, method, article or device. Without more limitations, the element defined by the statement “comprises a” does not exclude the presence of another identical element in the process, method, article or device comprising the element.

[0051] As known from the background, with the increase of the demand for file transmission using a batch data bus in the process of moving the relevant distributed core downward, the developers of the relevant project team need to configure more and more transmission files and tasks, but most of these to-be-transmitted files will be subscribed by different downstream systems, and the subscription is a very homogeneous task. Since the manual configuration of the file processing task is time-consuming and affects the relevant efficiency, it will lead to a decrease in the processing efficiency of the file transmission in the batch data bus. Therefore, how to improve the efficiency of the file transmission in the batch data bus is a problem that needs to be solved urgently by the present application.

[0052] To solve the above problems, the application discloses a file transmission processing method and device of a batch data bus. When the configurable requirements of each forwarding task are obtained, the data of an upstream system is sent to a downstream system through a batch data bus file transmission channel through the configurable requirements. The file transmission between the end-to-end can also be realized through the configurable file forwarding requirements. Each requirement only needs to configure the forwarding once to solve the file transmission requirements from the end-to-end. For the files stored in the batch data bus system, the files meeting the preset batch forwarding specification are batched and sent to the downstream system according to the preset batch forwarding rule and the batch data bus file transmission channel. In the process of batch sending the files meeting the preset batch forwarding specification to the downstream system, the files are copied, moved and backed up according to the forwarding requirement table of the file subscription, so as to ensure the continuity and reliability of the file operation, reduce the redundancy of repeated file sending, and improve the efficiency of file transmission in the batch data bus. The specific implementation is described in the following embodiments.

[0053] Reference Figure 1 As shown in the figure, the application discloses a file transmission processing method of a batch data bus, which mainly includes the following steps:

[0054] S101: When receiving a file stored in a storage area network (SAN), a control file is triggered to be generated. The control file is used to mark whether the file processing is completed.

[0055] In S101, when receiving a file uploaded by an upstream application system to a server local storage area network, a control file with the same name as the file stored in the storage area network is triggered to be generated.

[0056] Specifically, the file uploaded by the upstream application system to the server local SAN storage is a same-name file generated by the original file directory system with a control file suffix.ctl added. The control file is used to mark whether the file processing and forwarding are completed.

[0057] The control file with the suffix.ctl is a same-name suffix file of the batch data bus file transmission, which is used to judge the file transmission completion identifier.

[0058] S102: Scan all upstream original file directories, and if there are files in all upstream original file directories, select files meeting the preset batch forwarding specification from all upstream original file directories.

[0059] The current batch data bus file issuing process includes files with relatively fixed naming specification and files without fixed naming specification. Since data from the same upstream system is used by multiple downstream systems or branches, there is a need for fixed file name for different downstream subscription rules and data use requirements according to folder subscription. The.ctl file with the same name exists in the batch data bus file transmission system (EDP) batch data bus when the files are sent from various systems or branches to the batch data bus.

[0060] The relatively fixed naming specification file refers to the fixed file format of the batch data bus issuing, which is: province-city+system+table name+rule+date+.gz file format, such as 00-system name-table name-2G-20190111.gz. The length of the system table name is recommended to be no more than 50 characters. The separator must be separated by -. The table name cannot contain -, 00-system name-ABC_123-2G-20190111.gz is legal, and 00-system name-ABC-123-2G-20190111.gz is illegal.

[0061] The scanning of all upstream application system upstream original file directories is started through the monitoring script. If the original file directory contains files, the thread for processing files starts to work. The original files in the upstream directory that meet the relevant specifications are sent to the temporary (Temp) directory through the cut (MV) command, and the script program sends the files that do not meet the regulations to the Error error directory through the MV command.

[0062] The specific scanning of all upstream original file directories is shown in A1-A3.

[0063] A1: Scan all upstream original file directories through the monitoring script.

[0064] A2: If there are files in all upstream original file directories, notify the thread for processing files to start working.

[0065] A3: Through the thread for processing files, select the files consistent with the fixed file format issued by the batch data bus system node from all upstream original file directories, and determine the files consistent with the fixed file format issued by the batch data bus system node as the files meeting the preset batch forwarding specification.

[0066] The batch data bus system node is shown in Figure 2 .

[0067] Figure 2In the middle, by monitoring the script starts scanning all upstream application system upstream original file directory (ie, source element directory A, where the source file directory A includes source file directory 2 and source file directory 3, etc.), if the original file directory exists file start notification processing thread to start work, the upstream directory in accordance with the relevant specifications of the original file through the MV command to Temp temporary directory through the script program, and through the script program will not meet the pre-set batch forwarding specification of the file to the Error error directory;

[0068] After copying the files in the Temp temporary directory to the BackUp directory through the CP command to backup the files, record the backup time and the backup directory;

[0069] After the file backup is completed, copy the backup completed files to different forwarding downstream directories, and record the completion time of each downstream directory. The last time from the temporary directory MV to the target downstream directory, the temporary folder has no source file. Each CP or MV to the downstream directory will generate an encrypted file marked folder in the temporary directory through the encryption algorithm.

[0070] S103: When acquiring each forwarding task, batch send files that meet the pre-set batch forwarding specification to the downstream system through the pre-set batch forwarding rule and the batch data bus file transmission channel.

[0071] When acquiring each forwarding task, move the source file directory to the file in the to-be-forwarded directory for file forwarding corresponding to each forwarding task, and perform the actual forwarding operation to ensure that the files can be forwarded to the corresponding downstream directory in time.

[0072] After completing the forwarding, perform subsequent operations, including program logic, business logic, database, exception handling, etc.

[0073] Specifically, when acquiring each forwarding task, batch send files that meet the pre-set batch forwarding specification to the downstream system through the pre-set batch forwarding rule and the batch data bus file transmission channel. The process is shown as B1-B5.

[0074] B1: When acquiring each forwarding task, send the original file that meets the pre-set specification to the temporary directory through the MV command.

[0075] B2: Copy the files in the temporary directory to the BackUp directory through the CP command to backup the files, and record the backup time and the backup directory.

[0076] B3: After the file backup is completed, copy the files in the BackUp directory to different forwarding downstream directories, and record the completion time of each downstream directory.

[0077] B4: For each downstream directory, the completion time, the last time from the temporary directory MV to the target downstream directory within the completion time, until there is no source file in the temporary folder in the temporary directory, to complete the process of moving the source file directory to the file to be forwarded to the final.

[0078] For both CP and MV operations in A3, the source file name is operated first, and then a.ctl file is used to identify the completion of CP and MV operations for the files in the directory. If the temporary directory still has files, it means that the MV operation is not complete, and if the temporary directory has no files, it means that the file forwarding is complete.

[0079] According to the forwarding task batch, the files in the to-be-forwarded directory are forwarded to the corresponding downstream directory, so as to complete the process of sending the files meeting the preset batch forwarding specification to the downstream system.

[0080] In the process of forwarding the files in the to-be-forwarded directory to the corresponding downstream directory according to the forwarding task batch, if the machine is monitored to be down, the machine is restarted, the related files after the success are sent to the temporary directory, and the step of copying the files in the temporary directory to the BackUp directory through the CP command for file backup is performed.

[0081] After the process of forwarding the files in the to-be-forwarded directory to the corresponding downstream directory according to the forwarding task batch, the encrypted file is generated in the Temp temporary directory through the encryption algorithm, and the temp folder is marked by the encrypted file.

[0082] S104: When the files meeting the preset batch forwarding specification are all sent to the downstream system, the files batch forwarded to the downstream system are marked through the control file.

[0083] In order to facilitate the understanding of the process of batch data bus file transmission processing, examples are given for illustration:

[0084] For example, step 1: the file uploaded by the upstream application system to the server local SAN storage is automatically generated by the original file directory system. A file with the same name and a.ctl suffix is added to mark whether the file is processed. By monitoring the script, all upstream original file directories are scanned, and if there are files in the original file directory, the thread for processing the file is notified to start working. The original files in the upstream directory meeting the relevant specification are sent to the temporary directory through the MV command through the script program, and the files not meeting the specification are sent to the Error error directory through the MV command;

[0085] Step 2: File backup by copying the files in the temporary file directory to the BackUp directory through CP command, and record the backup time and the backup directory;

[0086] Step 3: After the file backup is completed, copy to different forwarding downstream directories respectively, and record the completion time of each downstream directory. The last time from the temporary directory MV to the target downstream directory, the temporary folder has no source file. Each CP or MV to the downstream directory will generate an encrypted file in the temporary directory to mark that folder through the encryption algorithm.

[0087] Step 4: File forwarding of each forwarding task for the source file directory finally moved to the to-be-forwarded directory, to perform the actual forwarding operation to ensure that the files can be forwarded to the corresponding downstream directory in time. The subsequent operation after completing the forwarding, such as program logic, business logic, database, exception handling, etc.

[0088] Step 5: For the CP and MV operations in step 2, the source file name is operated first, and then a.ctl file of the file is operated to identify that the directory file has completed the CP and MV operations. If the temporary directory still has files, it means that the MV operation has not been completed. If the temporary directory has no files, it means that the file forwarding is completed.

[0089] Step 6: For the file processing procedure after the machine is down due to program or other reasons and the machine is restarted successfully, the files in the temporary directory are re-MV to the source directory in step 2, and the subsequent judgment operation is performed on the basis of steps 1, 2, 3, and 4. The encrypted files in the temporary directory are decrypted, and the files that have been sent to the corresponding downstream directory are skipped and not processed. The continuity and reliability of file operation are ensured, and the redundancy of repeated file sending is reduced.

[0090] For the file processing procedure after the machine is down due to program or other reasons and the machine is restarted successfully, the files in the temporary directory are re-MV to the source directory in step 2, and the subsequent judgment operation is performed on the basis of steps 1, 2, 3, and 4. The encrypted files in the temporary directory are decrypted, and the files that have been sent to the corresponding downstream directory are skipped and not processed. The continuity and reliability of file operation are ensured, and the redundancy of repeated file sending is reduced.

[0091] The specific subsequent judgment operation in step 6 is shown in the following table. Figure 3

[0092] Figure 3 In the table, the Temp temporary file directory is obtained;

[0093] It is judged whether the Temp temporary file directory is empty; ​

[0094] If the Temp temporary file directory is empty, end the judgment operation process;

[0095] If the Temp temporary file directory is not empty, scan the source file directory and obtain the files in the source file directory;

[0096] Determine whether each file in the source file directory meets the specification requirements (i.e., the preset batch forwarding specification);

[0097] If the files in the source file directory do not meet the specification requirements, send the files that do not meet the specification requirements to the Error error directory through a script program;

[0098] If the files in the source file directory meet the specification requirements and there are three downstream systems (forwarding directory 1, forwarding directory 2, and forwarding directory 3) that require each file in the source file directory, perform steps ①, ②, ③, and ④ on each file in the source file directory that meets the specification requirements to implement forwarding the files to the corresponding three downstream systems (forwarding directory 1, forwarding directory 2, and forwarding directory 3); each forwarding directory corresponds to one requirement;

[0099] ①The files in the Temp temporary file directory are copied to the BackUp backup file directory in full; the BackUp backup file directory is used for file backup;

[0100] ②Determine whether the folder of the encrypted file has sent files, if not, according to the preset batch forwarding rule and batch data bus file transmission channel, batch copy the files that meet the preset batch forwarding specification to forwarding directory 1; if the files have been sent, end the process;

[0101] ③Determine whether the folder of the encrypted file has sent files, if not, according to the preset batch forwarding rule and batch data bus file transmission channel, batch copy the files that meet the preset batch forwarding specification to forwarding directory 2; if the files have been sent, end the process;

[0102] ④Determine whether the folder of the encrypted file has sent files, if not, according to the preset batch forwarding rule and batch data bus file transmission channel, batch move the files that meet the preset batch forwarding specification to forwarding directory 3; if the files have been sent, end the process.

[0103] It should be noted that there can be multiple forwarding directories, and the forwarding directory is used for specific processing of files. The Temp temporary file directory performs file operations in the order of ①, ②, ③, and ④, and the file of the last forwarding directory is a MV cut operation. Finally, the Temp temporary file directory is empty.

[0104] For example, if only one downstream system (such as forwarding directory 1) needs the files in the source file directory that meet the specification requirements, the files that meet the preset batch forwarding specification are batched MV to the forwarding directory 1 according to the preset batch forwarding rule and the batch data bus file transmission channel.

[0105] If two downstream systems (such as forwarding directory 1 and forwarding directory 2) need the files in the source file directory that meet the specification requirements, the files that meet the preset batch forwarding specification are batched CP to the forwarding directory 1, and the files in the source file directory that meet the specification requirements are MV to the forwarding directory 2 according to the preset batch forwarding rule and the batch data bus file transmission channel.

[0106] In order to facilitate the understanding of the process of the batch data bus file transmission processing, the following Figure 4 is described. Figure 4 The architecture diagram of the batch data bus file transmission processing is shown.

[0107] Figure 4 In the batch data bus file transmission processing architecture, the batch data bus file transmission processing architecture includes Server node 1 and Server node 2; system B, system C, system D, etc. are downstream systems; and F5 is a load balancing device.

[0108] The Server node 1 and the Server node 2 are load balanced in the process of the batch data bus file transmission processing through the load balancing device.

[0109] The batch data bus file transmission processing is consistent with the execution process and the execution principle of S101-S104 and steps 1-step 5 described above, and can be referred to, which will not be described here.

[0110] The application uses intelligent algorithm to identify files, has high accuracy, and has significant file processing effect; uses automatic script program to monitor files, ensures the timeliness of related file forwarding processing; supports quick recovery of related services after system or program downtime caused by force majeure; and supports configurable file operation, which is convenient for file forwarding transmission in various application scenarios.

[0111] The application sends the data of the upstream system to the downstream system through the batch data bus file transmission channel according to the configurable requirements. The file transmission between the head office and the branch can also be realized through the configurable file forwarding requirements. Each requirement only needs to be configured once to solve the file transmission requirements from end to end. For the files landed in the EDP, the files are CP, MV and backup according to the file forwarding requirement table to ensure the continuity and reliability of the file operation, reduce the redundancy of repeated file transmission, and improve the file transmission efficiency.

[0112] The beneficial effects of the embodiments of the present application are as follows: when the configurable requirements of each forwarding task are obtained, the data of the upstream system is sent to the downstream system through the batch data bus file transmission channel by the configurable requirements, and the file transmission between the end-to-end can also be realized by the configurable file forwarding requirements. Each requirement only needs to configure the forwarding once to solve the file transmission requirements from the end-to-end. For the files stored in the batch data bus system, the files meeting the preset batch forwarding specification are batch sent to the downstream system according to the preset batch forwarding rule and the batch data bus file transmission channel. In the process of batch sending the files meeting the preset batch forwarding specification to the downstream system, the files are copied, moved and backed up according to the forwarding requirement table of the file subscription, so as to ensure the continuity and reliability of the file operation, reduce the redundancy of repeated file sending, and improve the efficiency of file transmission in the batch data bus.

[0113] Based on the above embodiments Figure 1 The disclosed file transmission processing method of the batch data bus also discloses a file transmission processing device of the batch data bus, as shown in the figure, the file transmission processing device of the batch data bus comprises: Figure 5

[0114] The triggering unit 501 is configured to trigger the generation of a control file when receiving a file stored in the storage area network; the control file is used to mark whether the file processing is completed;

[0115] The scanning and selecting unit 502 is configured to scan all upstream original file directories, and select files meeting the preset batch forwarding specification from all upstream original file directories if there are files in all upstream original file directories;

[0116] The batch sending unit 503 is configured to batch send the files meeting the preset batch forwarding specification to the downstream system through the preset batch forwarding rule and the batch data bus file transmission channel when obtaining each forwarding task;

[0117] The marking unit 504 is configured to mark the files batch forwarded to the downstream system by the control file when the files meeting the preset batch forwarding specification are all sent to the downstream system.

[0118] Further, the triggering unit 501 is specifically configured to trigger the generation of a control file with the same name as the file stored in the storage area network when receiving the file stored in the storage area network uploaded to the server local by the upstream application system.

[0119] Further, the scanning and selecting unit 502 comprises:

[0120] The scanning module is configured to scan all upstream original file directories through a monitoring script;​

[0121] a notification module, configured to notify a processing file thread of work if there are files in all upstream original file directories;

[0122] a selection determination module, configured to select, by the processing file thread, files consistent with a fixed file format issued by the batch data bus from all upstream original file directories, and determine the files consistent with the fixed file format issued by the batch data bus system node as files meeting a preset batch forwarding specification.

[0123] Further, the batch sending unit 503 comprises:

[0124] a sending module, configured to send, by the processing file thread, the original files meeting the preset specification to a temporary directory through a cut command when the respective forwarding tasks are acquired;

[0125] a backup module, configured to copy the files in the temporary directory to a backup file directory through a copy command to perform file backup;

[0126] a copy recording module, configured to copy the files in the backup directory to different forwarding downstream directories respectively after the file backup is completed, and record a completion time of each downstream directory;

[0127] a moving module, configured to move, for the completion time of each downstream directory, the last time from the temporary directory to the target downstream directory within the completion time, until there is no source file in the temporary folder in the temporary directory, to complete the process of moving the files in the source file directory to the to-be-forwarded directory finally;

[0128] a forwarding module, configured to forward the files in the to-be-forwarded directory to respective corresponding downstream directories according to the batch forwarding tasks, to complete the process of sending the files meeting the preset batch forwarding specification to the downstream system.

[0129] Further, the file transmission processing device of the batch data bus further comprises:

[0130] a monitoring and restarting unit, configured to, in the process of forwarding the files in the to-be-forwarded directory to respective corresponding downstream directories according to the batch forwarding tasks, if a machine is monitored to be down, make the machine restart, send the related files after the success to the temporary directory, and perform the step of copying the files in the temporary directory to the backup directory through the copy command to perform file backup.

[0131] Further, the file transmission processing device of the batch data bus further comprises:

[0132] a recording unit, configured to record a backup time and a backup directory.

[0133] Further, the file transmission processing device of the batch data bus further comprises:

[0134] The generating marking unit is configured to generate an encrypted file in the temporary directory by using an encryption algorithm, and mark the temporary folder by using the encrypted file.

[0135] The application has the following beneficial effects: when the configurable requirements of each forwarding task are obtained, the data of the upstream system is sent to the downstream system through the batch data bus file transmission channel by using the configurable requirements, the file transmission between the end-to-end can also be realized by using the configurable file forwarding requirements, each requirement only needs to configure the forwarding once to solve the file transmission requirement from the end-to-end. For the file stored in the batch data bus system, the file meeting the preset batch forwarding specification is batch sent to the downstream system according to the preset batch forwarding rule and the batch data bus file transmission channel, the file is copied, moved and backed up according to the forwarding requirement table of the file subscription in the process of batch sending the file meeting the preset batch forwarding specification to the downstream system, the continuity and reliability of the file operation are ensured, the redundancy of repeated file sending is reduced, and the efficiency of the file transmission in the batch data bus is improved.

[0136] The application further provides a storage medium, which comprises stored instructions, wherein when the instructions are executed, the device where the storage medium is located performs the file transmission processing method of the batch data bus.

[0137] The application further provides an electronic device, and a structure diagram thereof is shown in the figure. Figure 6 The application further provides an electronic device, and a structure diagram thereof is shown in the figure.

[0138] For the foregoing method embodiments, in order to simply describe, they are all expressed as a series of action combinations, but the person skilled in the art should know that the application is not limited by the action sequence described, because according to the application, some steps can be performed in other sequences or simultaneously. Secondly, the person skilled in the art should know that the embodiments described in the specification all belong to preferred embodiments, and the actions and modules involved are not necessarily necessary for the application.

[0139] It should be noted that each of the embodiments in the specification is described in progressive mode, and each embodiment focuses on the difference from other embodiments. The same and similar parts between embodiments can be referred to each other. For system embodiments, since they are basically similar to method embodiments, the description is relatively simple, and the relevant parts can be referred to the part of the method embodiment.

[0140] The steps in the method of each embodiment of the present application can be adjusted, combined and reduced in sequence according to actual needs.

[0141] Finally, it should be noted that in this paper, relationship terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between the entities or operations.

[0142] The above description of the disclosed embodiments enables a person skilled in the art to implement or use the present application. Various modifications to the embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

[0143] The above is only the preferred embodiment of the present application. It should be noted that for ordinary skilled in the art, without departing from the principles of the present application, a number of improvements and refinements can be made, which should be considered as the protection scope of the present application.

Claims

1. A file transfer processing method of a bulk data bus, characterized by, The method comprises: When receiving a file stored in a storage area network, triggering generation of a control file; the control file is used for marking whether the file is processed; Scanning all upstream original file directories, and if there are files in all the upstream original file directories, selecting files meeting preset batch forwarding specifications from all the upstream original file directories; When obtaining each forwarding task, batch sending the files meeting the preset batch forwarding specifications to a downstream system through a preset batch forwarding rule and a batch data bus file transmission channel; When all the files meeting the preset batch forwarding specifications are sent to the downstream system, marking the files batch forwarded to the downstream system through the control file; When obtaining each forwarding task, batch sending the files meeting the preset batch forwarding specifications to a downstream system through a preset batch forwarding rule and a batch data bus file transmission channel; When obtaining each forwarding task, sending the original files meeting the preset specifications to a temporary directory through a cut command; Copying the files in the temporary directory to a backup file directory through a copy command to perform file backup; After the file backup is completed, copying the files in the backup directory to different forwarding downstream directories respectively, and recording the completion time of each downstream directory; For the completion time of each downstream directory, the last time of moving from the temporary directory to the target downstream directory within the completion time, until there is no source file in the temporary folder in the temporary directory, to complete the process of moving the source file directory to the file to be forwarded directory; According to the forwarding task batch, the file to be forwarded directory is forwarded to the corresponding downstream directory to complete the process of sending the file meeting the preset batch forwarding specification to the downstream system.

2. The method of claim 1, wherein, When receiving a file stored in a storage area network, triggering generation of a control file; the control file is used for marking whether the file is processed; When receiving a file stored in a storage area network, triggering generation of a control file; the control file is used for marking whether the file is processed; 3. The method of claim 1, wherein, Scanning all upstream original file directories, and if there are files in all the upstream original file directories, selecting files meeting preset batch forwarding specifications from all the upstream original file directories; Scanning all upstream original file directories through a monitoring script; If there are files in all the upstream original file directories, notifying a file processing thread to work; Through the file processing thread, selecting files consistent with a fixed file format issued by a batch data bus system node from all the upstream original file directories, and determining the files consistent with the fixed file format issued by the batch data bus system node as the files meeting the preset batch forwarding specifications.

4. The method of claim 1, wherein, Further comprising: In the process of forwarding the file to be forwarded directory to the corresponding downstream directory according to the forwarding task batch, if a machine is monitored to be down, restarting the machine, sending the related files after the success to a temporary directory, and performing the step of copying the files in the temporary directory to a backup directory through a copy command to perform file backup.

5. The method of claim 1, wherein, After the step of copying the files in the temporary directory to a backup directory through a copy command to perform file backup, further comprising: Record the time of backup and the directory of backup.

6. The method of claim 1, wherein, After the process of sending the files meeting the preset batch forwarding specification to the downstream system is completed, the method further comprises: An encrypted file is generated in the temporary directory through an encryption algorithm, and the temporary folder is marked through the encrypted file.

7. A file transfer processing apparatus of a bulk data bus, characterized by comprising: The device comprises: A triggering unit is configured to trigger generation of a control file when a file stored in a storage area network is received, the control file being used to mark whether the file is processed completely; A scanning and selecting unit is configured to scan all upstream original file directories, and select files meeting the preset batch forwarding specification from all the upstream original file directories if there are files in all the upstream original file directories; A batch sending unit is configured to send the files meeting the preset batch forwarding specification to a downstream system in batches through a preset batch forwarding rule and a batch data bus file transmission channel when each forwarding task is acquired; A marking unit is configured to mark the files forwarded to the downstream system in batches through the control file when the files meeting the preset batch forwarding specification are all sent to the downstream system. The batch sending unit comprises: A sending module is configured to send the original files meeting the preset specification to a temporary directory through a cut command when each forwarding task is acquired; A backup module is configured to copy the files in the temporary directory to a backup file directory through a copy command to perform file backup; A copy record module is configured to copy the files in the backup directory to different forwarding downstream directories respectively after the file backup is completed, and record the completion time of each downstream directory; A moving module is configured to move the files in the temporary directory to the target downstream directory in the last time within the completion time of each downstream directory until there is no source file in the temporary folder in the temporary directory, so as to complete the process of moving the files in the source file directory to the files in the to-be-forwarded directory finally. A forwarding module is configured to forward the files in the to-be-forwarded directory to respective corresponding downstream directories in batches according to forwarding tasks, so as to complete the process of sending the files meeting the preset batch forwarding specification to the downstream system.

8. The apparatus of claim 7, wherein, The triggering unit is specifically configured to: trigger generation of a control file with the same name as the file stored in the storage area network when the file stored in the storage area network uploaded to the server locally by an upstream application system is received.

9. The apparatus of claim 7, wherein, The scanning and selecting unit comprises: A scanning module is configured to scan all upstream original file directories through a monitoring script; A notification module is configured to notify a file processing thread to work if there are files in all the upstream original file directories; A selecting and determining module is configured to select files consistent with a fixed file format issued by a batch data bus system node from all the upstream original file directories through the file processing thread, and determine the files consistent with the fixed file format issued by the batch data bus system node as the files meeting the preset batch forwarding specification.

Images