File transmission processing method and device for batch data buses

Abstract

The invention discloses a file transmission processing method and device for a batch data bus, and the method comprises the steps: triggering the generation of a control file when a file stored in a storage area network is received, enabling the control file to be used for marking whether the file is processed or not, scanning all upstream original file directories, and if there is a file in all upstream original file directories, carrying out the scanning of all upstream original file directories; and selecting files conforming to a preset batch forwarding specification from all upstream original file directories, and when each forwarding task is obtained, sending the files conforming to the preset batch forwarding specification to a downstream system in batches through a preset batch forwarding rule and a batch data bus file transmission channel, and when all the files conforming to the preset batch forwarding specification are sent to the downstream system, marking the files forwarded to the downstream system in batches through the control file.

Description

Technical Field

[0001] The present application relates to the technical field of file transmission processing, and more specifically, to a file transmission processing method and device for a batch data bus. Background Art

[0002] As the demand for file transfer using batch data bus increases during the process of moving the relevant distributed core downward, the developers of the relevant project team need to configure more and more transfer files and tasks, but most of these files to be transferred will be subscribed by different downstream systems, and subscription is a very homogeneous task. Since the relevant manual configuration file processing tasks are time-consuming and affect the relevant efficiency, the processing efficiency of file transfer in the batch data bus will be reduced.

[0003] Therefore, how to improve the efficiency of file transmission in a batch data bus is a problem that needs to be solved urgently in this application. Summary of the invention

[0004] In view of this, the present application discloses a file transmission processing method and device for a batch data bus, aiming to reduce the redundancy of repeated file transmission to improve the efficiency of file transmission in the batch data bus.

[0005] In order to achieve the above purpose, the disclosed technical solution is as follows:

[0006] The first aspect of the present application discloses a file transmission processing method for a batch data bus, the method comprising:

[0007] When receiving a file stored in the storage area network, triggering the generation of a control file; the control file is used to mark whether the file has been processed;

[0008] Scan all upstream original file directories. If there are files in all upstream original file directories, select files that meet the preset batch forwarding specifications from all upstream original file directories.

[0009] When each forwarding task is obtained, the files that meet the preset batch forwarding specifications are sent to the downstream system in batches through the preset batch forwarding rules and batch data bus file transmission channel;

[0010] When all the files that meet the preset batch forwarding specification are sent to the downstream system, the files that are batch forwarded to the downstream system are marked through the control file.

[0011] Preferably, when receiving a file stored in a storage area network, triggering the generation of a control file includes:

[0012] When a file uploaded by an upstream application system to a local storage area network storage of a server is received, a control file having the same name as the file stored in the storage area network is triggered to be generated.

[0013] Preferably, the step of scanning all upstream original file directories and selecting files that meet a preset batch forwarding specification from all upstream original file directories if files exist in all upstream original file directories includes:

[0014] Scan all upstream raw file directories through monitoring scripts;

[0015] If the file exists in all upstream original file directories, notify the file processing thread to work;

[0016] Through the file processing thread, files consistent with the fixed file format sent by the batch data bus are selected from all upstream original file directories, and files consistent with the fixed file format sent by the batch data bus system node are determined as files that meet the preset batch forwarding specification.

[0017] Preferably, when each forwarding task is obtained, the files that meet the preset batch forwarding specification are sent in batches to the downstream system through the preset batch forwarding rules and the batch data bus file transmission channel, including:

[0018] When each forwarding task is obtained, the original file that meets the preset specifications is sent to the temporary directory through the cut command;

[0019] Use the copy command to copy the files in the temporary directory to the backup file directory for file backup;

[0020] After the file backup is completed, the files in the backup directory are copied to different forwarding downstream directories respectively, and the completion time of each downstream directory is recorded;

[0021] For each downstream directory completion time, the last move from the temporary directory to the target downstream directory within the completion time, until there is no source file in the temporary folder in the temporary directory, so as to complete the process of finally moving the source file directory to the to-be-forwarded directory;

[0022] The files in the to-be-forwarded directory are forwarded to their corresponding downstream directories in batches according to the forwarding task, so as to complete the process of sending the files that meet the preset batch forwarding specifications to the downstream system.

[0023] Preferably, it also includes:

[0024] In the process of forwarding the files in the to-be-forwarded directory to their corresponding downstream directories in batches according to the forwarding task, if the machine is detected to be down, the machine is restarted, the relevant files are sent to the temporary directory after success, and the files in the temporary directory are copied to the backup directory through the copy command for file backup.

[0025] Preferably, after copying the files in the temporary directory to the backup directory by using the copy command to perform file backup, the method further includes:

[0026] Record the backup time and backup directory.

[0027] Preferably, after forwarding the files in the to-be-forwarded directory to the corresponding downstream directories in batches according to the forwarding task to complete the process of sending the files that meet the preset batch forwarding specification to the downstream system, the method further includes:

[0028] An encrypted file is generated in the temporary directory by an encryption algorithm, and the temporary folder is marked by the encrypted file.

[0029] The second aspect of the present application discloses a file transmission processing device for a batch data bus, the device comprising:

[0030] A trigger unit, used to trigger the generation of a control file when receiving a file stored in the storage area network; the control file is used to mark whether the file has been processed;

[0031] A scanning and selecting unit is used to scan all upstream original file directories, and if there are files in all upstream original file directories, select files that meet the preset batch forwarding specifications from all upstream original file directories;

[0032] A batch sending unit is used to send files that meet the preset batch forwarding specifications to the downstream system in batches through preset batch forwarding rules and batch data bus file transmission channels when each forwarding task is obtained;

[0033] The marking unit is used to mark the files forwarded in batches to the downstream system through the control file when all the files that meet the preset batch forwarding specification are sent to the downstream system.

[0034] Preferably, the trigger unit is specifically used for:

[0035] When a file uploaded by an upstream application system to a local storage area network storage of a server is received, a control file having the same name as the file stored in the storage area network is triggered to be generated.

[0036] Preferably, the scanning and selecting unit comprises:

[0037] Scanning module, used to scan all upstream original file directories through monitoring scripts;

[0038] Notification module, used to notify the file processing thread if there are files in all upstream original file directories;

[0039] The selection and determination module is used to select files consistent with the fixed file format sent by the batch data bus from all upstream original file directories through the file processing thread, and determine the files consistent with the fixed file format sent by the batch data bus system node as files that meet the preset batch forwarding specifications.

[0040] It can be known from the above technical scheme that the present application discloses a file transmission processing method and device of a batch data bus. When a file stored in a storage area network is received, a control file is triggered to be generated. The control file is used to mark whether the file processing is completed. All upstream original file directories are scanned. If there are files in all upstream original file directories, files that meet the preset batch forwarding specifications are selected from all upstream original file directories. When each forwarding task is obtained, the files that meet the preset batch forwarding specifications are sent to the downstream system in batches through the preset batch forwarding rules and the batch data bus file transmission channel. When all the files that meet the preset batch forwarding specifications are sent to the downstream system, the files that are batch forwarded to the downstream system are marked through the control file.

[0041] Through the above scheme, when the configurable requirements of each forwarding task are obtained, the data of the upstream system is sent to the downstream system through the batch data bus file transmission channel through the configurable requirements. For end-to-end file transmission, the configurable file forwarding requirements can also be used. Each requirement only needs to be configured once to solve the end-to-end file transmission requirements. For files stored in the batch data bus system, according to the preset batch forwarding rules and batch data bus file transmission channels, the files that meet the preset batch forwarding specifications are sent to the downstream system in batches. In the process of sending the files that meet the preset batch forwarding specifications to the downstream system in batches, the files are copied, moved and backed up according to the forwarding requirement table subscribed by the files, ensuring the continuity and reliability of file operations, reducing the redundancy of repeated file transmission, and improving the efficiency of file transmission in the batch data bus. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0043] Figure 1 A flowchart of a file transmission processing method of a batch data bus disclosed in an embodiment of the present application;

[0044] Figure 2 A schematic diagram of a file processing task disclosed in an embodiment of the present application;

[0045] Figure 3 A schematic diagram of the file processing judgment process disclosed in the embodiment of the present application;

[0046] Figure 4 This is an architectural diagram of file transfer processing of a batch data bus disclosed in an embodiment of the present application;

[0047] Figure 5 A schematic diagram of the structure of a file transmission processing device for a batch data bus disclosed in an embodiment of the present application;

[0048] Figure 6 This is a schematic diagram of the structure of an electronic device disclosed in an embodiment of the present application. DETAILED DESCRIPTION

[0049] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0050] In this application, the terms "comprises", "comprising" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprising a ..." does not exclude the presence of other identical elements in the process, method, article or device comprising the element.

[0051] As can be seen from the background technology, with the increase in the demand for file transfer using the batch data bus during the process of moving the relevant distributed core downward, the demand for the transfer files and tasks that the developers of the relevant project team need to configure is also increasing. However, most of these files to be transferred will be subscribed by different downstream systems, and subscription is a very homogeneous task. Since the relevant manual configuration file processing tasks are time-consuming and affect the relevant efficiency, the processing efficiency of the file transfer in the batch data bus will be reduced. Therefore, how to improve the efficiency of file transfer in the batch data bus is a problem that needs to be solved urgently in this application.

[0052] In order to solve the above problems, the present application discloses a file transfer processing method and device for a batch data bus. When the configurable requirements of each forwarding task are obtained, the data of the upstream system is sent to the downstream system through the batch data bus file transfer channel through the configurable requirements. For end-to-end file transfer, the configurable file forwarding requirements can also be used. Each requirement only needs to be configured once to solve the end-to-end file transfer requirements. For files stored in the batch data bus system, according to the preset batch forwarding rules and the batch data bus file transfer channel, the files that meet the preset batch forwarding specifications are sent to the downstream system in batches. In the process of sending the files that meet the preset batch forwarding specifications to the downstream system in batches, the files are copied, moved and backed up according to the forwarding requirement table subscribed by the files, thereby ensuring the continuity and reliability of file operations, reducing the redundancy of repeated file sending, and improving the efficiency of file transmission in the batch data bus. The specific implementation method is specifically described through the following embodiments.

[0053] refer to Figure 1 As shown, a file transmission processing method of a batch data bus disclosed in an embodiment of the present application is shown, and the file transmission processing method of the batch data bus mainly includes the following steps:

[0054] S101: When a file stored in a storage area network (SAN) is received, a control file is generated; the control file is used to mark whether the file has been processed.

[0055] In S101, when a file uploaded by an upstream application system to a local storage area network storage of a server is received, a control file having the same name as the file stored in the storage area network is generated.

[0056] Specifically, for files uploaded by the upstream application system to the local SAN storage of the server, the original file directory system will automatically generate a control file with the same name and the suffix .ctl. The control file is used to mark whether the file has been processed and forwarded.

[0057] The control file with the suffix .ctl is the file with the same suffix name as the data bus file transmission, and is used to determine the file transfer completion mark.

[0058] S102: Scan all upstream original file directories, and if there are files in all upstream original file directories, select files that meet preset batch forwarding specifications from all upstream original file directories.

[0059] In the current batch data bus file distribution process, the file names include files with relatively fixed naming conventions and files with non-fixed naming conventions. Since the data of the same upstream system may have related data usage requirements in multiple downstream systems or branches, there are requirements for fixed file names and data usage requirements for different downstream subscription rules, and files sent from various systems or branches to the batch data bus file transfer system (EDP) will have .ctl files with the same file name.

[0060] Among them, the relatively fixed naming standard file refers to the fixed file format issued by the batch data bus: province + system + table name + rule + date + .gz file format, such as 00-system name-table name-2G-20190111.gz; the length of the system table name is recommended to be no more than 50 characters. The delimiter must be separated by -, and the table name cannot contain -. 00-system name-ABC_123-2G-20190111.gz is legal, and 00-system name-ABC-123-2G-20190111.gz is illegal.

[0061] Start scanning the upstream original file directories of all upstream application systems through the monitoring script. If there are files in the original file directories, start notifying the thread that processes the files to start working. Send the original files in the upstream directory that meet the relevant specifications to the temporary (Temp) directory through the Cut (MV) command, and send the files that do not meet the regulations to the Error directory through the script program using the MV command.

[0062] Specifically, all upstream original file directories are scanned. If files exist in all upstream original file directories, the process of selecting files that meet the preset batch forwarding specification from all upstream original file directories is shown in A1-A3.

[0063] A1: Scan all upstream original file directories through monitoring scripts.

[0064] A2: If there are files in all upstream original file directories, notify the file processing thread to start working.

[0065] A3: By processing the file thread, files consistent with the fixed file format sent by the batch data bus are selected from all upstream original file directories, and files consistent with the fixed file format sent by the batch data bus system node are determined as files that meet the preset batch forwarding specifications.

[0066] Bulk data bus system nodes such as Figure 2 shown.

[0067] Figure 2In the process, the upstream original file directories (i.e., source component directory A, where source file directory A includes source file directory 2 and source file directory 3, etc.) of all upstream application systems are scanned through the monitoring script. If there are files in the original file directory, the thread processing the files is notified to start working. The original files in the upstream directory that meet the relevant specifications are sent to the Temp temporary directory through the MV command through the script program, and the files that do not meet the preset batch forwarding specifications are sent to the Error error directory through the script program.

[0068] After using the CP command to copy files in the Temp directory to the BackUp directory for file backup, record the backup time and backup directory;

[0069] After the file backup is completed, the backed-up files are copied to different forwarding downstream directories, and the completion time of each downstream directory is recorded. The last time from the temporary directory MV to the target downstream directory, there is no source file in the temporary folder. Each time CP or MV is transferred to the downstream directory, an encrypted file marked folder will be generated in the temporary directory through the encryption algorithm.

[0070] S103: When each forwarding task is obtained, the files that meet the preset batch forwarding specification are sent in batches to the downstream system through the preset batch forwarding rules and the batch data bus file transmission channel.

[0071] When each forwarding task is obtained, the source file directory is finally moved to the file to be forwarded directory to perform file forwarding corresponding to each forwarding task, and the actual forwarding operation is performed to ensure that the files can be forwarded to the corresponding downstream directory in time.

[0072] After the forwarding is completed, subsequent operations are performed, including program logic, business logic, database, exception handling, etc.

[0073] Specifically, when each forwarding task is obtained, the process of sending files that meet the preset batch forwarding specification to the downstream system in batches through the preset batch forwarding rules and the batch data bus file transmission channel is shown in B1-B5.

[0074] B1: When each forwarding task is obtained, the original file that meets the preset specifications is sent to the temporary directory through the MV command.

[0075] B2: Use the CP command to copy the files in the temporary directory to the BackUp directory for file backup, and record the backup time and backup directory.

[0076] B3: After the file backup is completed, the files in the BackUp directory are copied to different forwarding downstream directories respectively, and the completion time of each downstream directory is recorded.

[0077] B4: For the completion time of each downstream directory, the last move from the temporary directory MV to the target downstream directory within the completion time is completed until there is no source file in the temporary folder in the temporary directory, so as to complete the process of finally moving the files from the source file directory to the directory to be forwarded.

[0078] For CP and MV operations in A3, the source file name is operated first and then the .ctl file of the file is operated to indicate that the files in the directory have completed CP and MV operations. If there are still files in the temporary directory, it means that the MV operation is not completed. If there are no files in the temporary directory, it means that the file forwarding is completed.

[0079] The files in the to-be-forwarded directory are forwarded to their corresponding downstream directories in batches according to the forwarding task, so as to complete the process of sending the files that meet the preset batch forwarding specifications to the downstream system.

[0080] In the process of forwarding the files in the to-be-forwarded directory to their respective corresponding downstream directories in batches according to the forwarding task, if the machine crashes, the machine is restarted, and the related files are sent to the temporary directory after success, and the step of copying the files in the temporary directory to the BackUp directory for file backup is executed through the CP command.

[0081] After forwarding the files in the to-be-forwarded directory to their respective corresponding downstream directories in batches according to the forwarding task to complete the process of sending the files that meet the preset batch forwarding specifications to the downstream system, an encrypted file is generated in the Temp temporary directory through an encryption algorithm, and the temp folder is marked with the encrypted file.

[0082] S104: When all files that meet the preset batch forwarding specification are sent to the downstream system, the files that are batch forwarded to the downstream system are marked through a control file.

[0083] In order to facilitate the understanding of the file transfer process of the bulk data bus, an example is given here:

[0084] For example, step 1: For files uploaded by the upstream application system to the local SAN storage of the server, the original file directory system will automatically generate a file with the same name and the suffix .ctl to mark whether the file has been processed. Start scanning all upstream original file directories through the monitoring script. If there are files in the original file directory, start notifying the thread that processes the file to start working. Send the original files that meet the relevant specifications in the upstream directory to the temporary directory through the MV command through the script program, and send the MV command to the Error directory for files that do not meet the regulations;

[0085] Step 2: Use the CP command to copy the files in the temporary file directory to the BackUp directory for file backup, and record the backup time and backup directory;

[0086] Step 3: After the file backup is completed, it is copied to different forwarding downstream directories, and the completion time of each downstream directory is recorded. The last time from the temporary directory MV to the target downstream directory, there is no source file in the temporary folder. Every time CP or MV goes to the downstream directory, an encrypted file will be generated in the temporary directory through the encryption algorithm to mark that folder.

[0087] Step 4: Perform file forwarding for each forwarding task for the files that are finally moved from the source file directory to the directory to be forwarded, and perform actual forwarding operations to ensure that the files can be forwarded to the corresponding downstream directory in a timely manner. Complete the subsequent operations after forwarding, such as program logic, business logic, database, exception handling, etc.

[0088] Step 5: For the CP and MV operations in step 2, the source file name is operated first and then the .ctl file of the file is operated to indicate that the files in the directory have completed CP and MV operations. If there are still files in the temporary directory, it means that the MV operation is not completed. If there are no files in the temporary directory, it means that the file forwarding is completed.

[0089] Step 6: When the machine crashes due to programs or other reasons, the processing flow of the related files after the machine and the successful restart will re-MV the files in the temporary directory to the source directory in step 2, and re-operate steps 1, 2, 3, and 4. The relevant subsequent judgment operations will be performed, and the encrypted files in the temporary directory will be decrypted. The files that have been sent to the corresponding downstream directory will be skipped and no longer processed. The continuity and reliability of file operations are ensured, and the redundancy of repeated file sending is reduced.

[0090] Among them, when the machine crashes due to programs or other reasons, the processing flow of related files after the machine and the successful restart will re-MV the files in the temporary directory to the source directory in step 2, and re-perform related subsequent judgment operations based on steps 1, 2, 3, and 4, and decrypt the encrypted files in the temporary directory. The files that have been sent to the corresponding downstream directory will be skipped and no longer processed. The continuity and reliability of file operations are ensured, and the redundancy of repeated file sending is reduced.

[0091] The relevant subsequent judgment operations in step 6 are as follows: Figure 3 shown.

[0092] Figure 3 , get the Temp temporary file directory;

[0093] Determine whether the Temp temporary file directory is empty;

[0094] If the Temp temporary file directory is empty, the judgment operation process ends;

[0095] If the Temp file directory is not empty, scan the source file directory and obtain the files in the source file directory;

[0096] Determine whether each file in the source file directory meets the specification requirements (i.e., the preset batch forwarding specifications);

[0097] If the files in the source file directory do not meet the specification requirements, the script program will send the files that do not meet the specification requirements to the Error directory;

[0098] If the files in the source file directory meet the specification requirements, and there are three downstream systems (forwarding directory 1, forwarding directory 2, and forwarding directory 3) that need the files in the source file directory, perform steps ①, ②, ③, and ④ on the files in the source file directory that meet the specification requirements in batches to forward the files to the corresponding three downstream systems (forwarding directory 1, forwarding directory 2, and forwarding directory 3); each forwarding directory corresponds to one requirement;

[0099] ① The files in the Temp directory will be fully copied to the BackUp directory. The BackUp directory is used for file backup.

[0100] ② Determine whether the folder of the encrypted file has been sent by the CP. If not, the files that meet the preset batch forwarding specifications are batched to the forwarding directory 1 according to the preset batch forwarding rules and batch data bus file transmission channel; if the files have been sent by the CP, the process ends;

[0101] ③ Determine whether the folder of the encrypted file has been sent by the CP. If not, the files that meet the preset batch forwarding specifications are batched to the forwarding directory 2 according to the preset batch forwarding rules and batch data bus file transmission channel; if the files have been sent by the CP, the process ends;

[0102] ④ Determine whether the folder of the encrypted file has been sent by CP. If not, according to the preset batch forwarding rules and batch data bus file transmission channel, batch MV the files that meet the preset batch forwarding specifications to forwarding directory 3; if the CP has sent files, end the process.

[0103] It should be noted that there may be multiple forwarding directories. The forwarding directories are used for specific file processing. The Temp temporary file directory performs file operations in the order of ①, ②, ③, and ④. The file in the last forwarding directory is the MV cut operation. Finally, the file in the Temp temporary file directory is empty.

[0104] For example, if only one downstream system (such as forwarding directory 1) needs files that meet the specification requirements in the source file directory, then according to the preset batch forwarding rules and batch data bus file transmission channel, the files that meet the preset batch forwarding specifications are batch MVed to forwarding directory 1.

[0105] If there are two downstream systems (such as forwarding directory 1 and forwarding directory 2) that need files in the source file directory that meet the specification requirements, then according to the preset batch forwarding rules and batch data bus file transmission channel, the files that meet the preset batch forwarding specifications are batch CP to forwarding directory 1, and the files that meet the specification requirements in the source file directory are MVed to forwarding directory 2.

[0106] In order to facilitate the understanding of the bulk data bus file transfer process, combined with Figure 4 Provide explanation. Figure 4 An architectural diagram showing file transfer processing over a bulk data bus.

[0107] Figure 4 In the example, the file transfer processing architecture of the batch data bus includes Server Node 1 and Server Node 2; System B, System C, System D, etc. are all downstream systems; and F5 is a load balancing device.

[0108] The load balancing device enables the server node 1 and the server node 2 to achieve load balancing during the file transmission process of the batch data bus.

[0109] The file transfer processing of the batch data bus is consistent with the execution process and execution principle of the above S101-S104, and steps 1-5, which can be referred to and will not be repeated here.

[0110] This application uses an intelligent algorithm to identify files with high accuracy and significant file processing effects; it uses an automated script program to monitor files to ensure the timeliness of related file forwarding and processing; it supports rapid recovery of related services after system or program crashes due to force majeure; it supports configurable file operations to facilitate file forwarding and transmission in a variety of application scenarios.

[0111] This application uses configurable requirements to send data from the upstream system to the downstream system through a batch data bus file transfer channel. For file transfers between the head office and branches, configurable file forwarding requirements can also be used. Each requirement only needs to be configured once to solve the end-to-end file transfer requirements. For files that have landed in the EDP, CP and MV are performed on the files according to the forwarding requirement table of the file subscription, and the files are backed up, ensuring the continuity and reliability of file operations, reducing the redundancy of repeated file sending, and achieving the effect of improving file transfer efficiency.

[0112] The beneficial effects of the embodiments of the present application are as follows: when the configurable requirements of each forwarding task are obtained, the data of the upstream system is sent to the downstream system through the batch data bus file transmission channel through the configurable requirements. The end-to-end file transmission can also be solved through the configurable file forwarding requirements. Each requirement only needs to be configured once to solve the end-to-end file transmission requirements. For files stored in the batch data bus system, according to the preset batch forwarding rules and the batch data bus file transmission channel, the files that meet the preset batch forwarding specifications are sent to the downstream system in batches. In the process of sending the files that meet the preset batch forwarding specifications to the downstream system in batches, the files are copied, moved and backed up according to the forwarding requirement table of the file subscription, thereby ensuring the continuity and reliability of the file operation and reducing the redundancy of repeated file transmission, so as to improve the efficiency of file transmission in the batch data bus.

[0113] Based on the above embodiments Figure 1 A method for processing file transmission via a batch data bus is disclosed. The present application also discloses a device for processing file transmission via a batch data bus. Figure 5 As shown, the file transmission processing device of the batch data bus includes:

[0114] A trigger unit 501 is used to trigger the generation of a control file when receiving a file stored in a storage area network; the control file is used to mark whether the file has been processed;

[0115] The scanning and selecting unit 502 is used to scan all upstream original file directories, and if there are files in all upstream original file directories, select files that meet the preset batch forwarding specification from all upstream original file directories;

[0116] The batch sending unit 503 is used to send the files that meet the preset batch forwarding specification to the downstream system in batches through the preset batch forwarding rules and the batch data bus file transmission channel when each forwarding task is obtained;

[0117] The marking unit 504 is used to mark the files forwarded in batches to the downstream system through a control file when all the files that meet the preset batch forwarding specification are sent to the downstream system.

[0118] Furthermore, the trigger unit 501 is specifically configured to trigger the generation of a control file having the same name as the file stored in the storage area network when receiving a file uploaded by an upstream application system to the local storage area network storage of the server.

[0119] Furthermore, the scanning and selecting unit 502 includes:

[0120] Scanning module, used to scan all upstream original file directories through monitoring scripts;

[0121] Notification module, used to notify the file processing thread if there are files in all upstream original file directories;

[0122] The selection and determination module is used to select files that are consistent with the fixed file format sent by the batch data bus from all upstream original file directories by processing file threads, and determine the files that are consistent with the fixed file format sent by the batch data bus system node as files that meet the preset batch forwarding specifications.

[0123] Furthermore, the batch sending unit 503 includes:

[0124] The sending module is used to send the original files that meet the preset specifications to the temporary directory through the cut command when each forwarding task is obtained;

[0125] A backup module is used to copy files in a temporary directory to a backup file directory through a copy command for file backup;

[0126] A copy record module is used to copy the files in the backup directory to different forwarding downstream directories after the file backup is completed, and record the completion time of each downstream directory;

[0127] A moving module is used to move files from the temporary directory to the target downstream directory for the last time within the completion time for each downstream directory until there are no source files in the temporary folder in the temporary directory, so as to complete the process of finally moving the files from the source file directory to the to-be-forwarded directory;

[0128] The forwarding module is used to forward the files in the to-be-forwarded directory to their corresponding downstream directories in batches according to the forwarding task, so as to complete the process of sending the files that meet the preset batch forwarding specifications to the downstream system.

[0129] Furthermore, the file transmission processing device of the batch data bus also includes:

[0130] The monitoring and restart unit is used to forward the files in the to-be-forwarded directory to their respective corresponding downstream directories in batches according to the forwarding task. If a machine crash is detected, the machine will be restarted, the relevant files will be sent to a temporary directory after success, and the step of copying the files in the temporary directory to the backup directory through the copy command for file backup will be executed.

[0131] Furthermore, the file transmission processing device of the batch data bus also includes:

[0132] Recording unit, used to record the backup time and backup directory.

[0133] Furthermore, the file transmission processing device of the batch data bus also includes:

[0134] The marking generating unit is used for generating an encrypted file in a temporary directory by using an encryption algorithm, and marking the temporary folder by using the encrypted file.

[0135] The beneficial effects of the embodiments of the present application are as follows: when the configurable requirements of each forwarding task are obtained, the data of the upstream system is sent to the downstream system through the batch data bus file transmission channel through the configurable requirements. The end-to-end file transmission can also be solved through the configurable file forwarding requirements. Each requirement only needs to be configured once to solve the end-to-end file transmission requirements. For files stored in the batch data bus system, according to the preset batch forwarding rules and the batch data bus file transmission channel, the files that meet the preset batch forwarding specifications are sent to the downstream system in batches. In the process of sending the files that meet the preset batch forwarding specifications to the downstream system in batches, the files are copied, moved and backed up according to the forwarding requirement table of the file subscription, thereby ensuring the continuity and reliability of the file operation and reducing the redundancy of repeated file transmission, so as to improve the efficiency of file transmission in the batch data bus.

[0136] An embodiment of the present application further provides a storage medium, which includes stored instructions, wherein when the instructions are executed, the device where the storage medium is located is controlled to execute the file transfer processing method of the batch data bus as described above.

[0137] The present application also provides an electronic device, whose structural diagram is as follows: Figure 6 As shown, it specifically includes a memory 601 and one or more instructions 602, wherein the one or more instructions 602 are stored in the memory 601 and are configured to be executed by one or more processors 603 to execute the one or more instructions 602 to perform the above-mentioned file transfer processing method of the bulk data bus.

[0138] For the aforementioned method embodiments, for the sake of simplicity, they are all described as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the order of the actions described, because according to the present application, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.

[0139] It should be noted that each embodiment in this specification is described in a progressive manner, and each embodiment focuses on the differences from other embodiments, and the same or similar parts between the embodiments can be referred to each other. For system embodiments, since they are basically similar to method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0140] The steps in the methods of the embodiments of the present application can be adjusted in order, combined, and deleted according to actual needs.

[0141] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are merely used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations.

[0142] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

[0143] The above is only a preferred implementation of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A file transfer processing method for a batch data bus, characterized in that: The method comprises: When receiving a file stored in the storage area network, triggering the generation of a control file; the control file is used to mark whether the file has been processed; Scan all upstream original file directories. If there are files in all upstream original file directories, select files that meet the preset batch forwarding specifications from all upstream original file directories. When each forwarding task is obtained, the files that meet the preset batch forwarding specifications are sent to the downstream system in batches through the preset batch forwarding rules and batch data bus file transmission channel; When all the files that meet the preset batch forwarding specification are sent to the downstream system, the files that are batch forwarded to the downstream system are marked through the control file.

2. The method according to claim 1, characterized in that When receiving a file stored in the storage area network, triggering the generation of a control file includes: When a file uploaded by an upstream application system to a local storage area network storage of a server is received, a control file having the same name as the file stored in the storage area network is triggered to be generated.

3. The method according to claim 1, characterized in that The scanning of all upstream original file directories, if files exist in all upstream original file directories, selecting files that meet the preset batch forwarding specification from all upstream original file directories, includes: Scan all upstream raw file directories through monitoring scripts; If the file exists in all upstream original file directories, notify the file processing thread to work; Through the file processing thread, files consistent with the fixed file format sent by the batch data bus are selected from all upstream original file directories, and files consistent with the fixed file format sent by the batch data bus system node are determined as files that meet the preset batch forwarding specification.

4. The method according to claim 1, characterized in that: When each forwarding task is obtained, the files that meet the preset batch forwarding specification are sent in batches to the downstream system through the preset batch forwarding rules and the batch data bus file transmission channel, including: When each forwarding task is obtained, the original file that meets the preset specifications is sent to the temporary directory through the cut command; Use the copy command to copy the files in the temporary directory to the backup file directory for file backup; After the file backup is completed, the files in the backup directory are copied to different forwarding downstream directories respectively, and the completion time of each downstream directory is recorded; For each downstream directory completion time, the last move from the temporary directory to the target downstream directory within the completion time, until there is no source file in the temporary folder in the temporary directory, so as to complete the process of finally moving the source file directory to the to-be-forwarded directory; The files in the to-be-forwarded directory are forwarded to their corresponding downstream directories in batches according to the forwarding task, so as to complete the process of sending the files that meet the preset batch forwarding specifications to the downstream system.

5. The method according to claim 4, characterized in that Also includes: In the process of forwarding the files in the to-be-forwarded directory to their corresponding downstream directories in batches according to the forwarding task, if the machine is detected to be down, the machine is restarted, the relevant files are sent to the temporary directory after success, and the files in the temporary directory are copied to the backup directory through the copy command for file backup.

6. The method according to claim 4, characterized in that After the files in the temporary directory are copied to the backup directory by the copy command to perform file backup, the method further includes: Record the backup time and backup directory.

7. The method according to claim 4, characterized in that After forwarding the files in the to-be-forwarded directory to the corresponding downstream directories in batches according to the forwarding task to complete the process of sending the files that meet the preset batch forwarding specification to the downstream system, the process further includes: An encrypted file is generated in the temporary directory by an encryption algorithm, and the temporary folder is marked by the encrypted file.

8. A file transfer processing device for a batch data bus, characterized in that: The device comprises: A trigger unit, used to trigger the generation of a control file when receiving a file stored in the storage area network; the control file is used to mark whether the file has been processed; A scanning and selecting unit is used to scan all upstream original file directories, and if there are files in all upstream original file directories, select files that meet the preset batch forwarding specifications from all upstream original file directories; A batch sending unit is used to send files that meet the preset batch forwarding specifications to the downstream system in batches through preset batch forwarding rules and batch data bus file transmission channels when each forwarding task is obtained; The marking unit is used to mark the files forwarded in batches to the downstream system through the control file when all the files that meet the preset batch forwarding specification are sent to the downstream system.

9. The device according to claim 8, characterized in that The trigger unit is specifically used for: When a file uploaded by an upstream application system to a local storage area network storage of a server is received, a control file having the same name as the file stored in the storage area network is triggered to be generated.

10. The device according to claim 8, characterized in that The scanning and selecting unit comprises: Scanning module, used to scan all upstream original file directories through monitoring scripts; Notification module, used to notify the file processing thread if there are files in all upstream original file directories; The selection and determination module is used to select files consistent with the fixed file format sent by the batch data bus from all upstream original file directories through the file processing thread, and determine the files consistent with the fixed file format sent by the batch data bus system node as files that meet the preset batch forwarding specifications.

Images