An unstructured file transmission method based on Flink

Abstract

The present application relates to the technical field of data processing, and provide a kind of based on Flink's unstructured file transmission method, the method of the present application includes: reading source file configuration information, obtains file quantity and file synchronization execution thread number;According to the file quantity and file synchronization execution thread number obtained distribution file transmission task;Check target file, create source file server connection for the target file that passes through check, query corresponding source file;Create target file server connection for the source file obtained by query, write target file into corresponding target file path.Under the unstructured file transmission method based on Flink according to the exemplary embodiment of the present application, it can reduce deployment difficulty, reduce maintenance difficulty, improve system stability, avoid file duplication transmission, repeated read-write, reduce additional read-write consumption of disk, reduce the time consumed by the whole synchronization.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, and in particular to a non-structured file transmission method based on Flink. BACKGROUND

[0002] Apache Flink (hereinafter referred to as Flink) is an open source stream processing framework developed by Apache Software Foundation, and its core is a distributed stream data flow engine written in Java and Scala. Flink executes any stream data program in a data-parallel and pipelined manner, and the pipelined runtime system of Flink can execute batch processing and stream processing programs. In addition, the runtime of Flink itself also supports the execution of iterative algorithms. Due to the characteristics of high throughput, low delay and the ability to execute any stream data program in a data-parallel and pipelined manner, Flink is widely used in the collection, transmission, synchronization and analysis of big data scenes, and in particular in the structured data synchronization and transmission scene, Flink has become the mainstream technical selection scheme due to its advantages.

[0003] However, in production, there are often scenes where structured data and unstructured data need to be synchronized. The traditional synchronization method is to synchronize structured and unstructured data separately, structured data is synchronized through Flink, and unstructured data is often synchronized through the deployment of FTP / SFTP tools or through FTP / SFTP scripts. The two synchronization processes are independent of each other and cannot be managed uniformly, and the deployment of FTP / SFTP tools, through FTP / SFTP scripts and other methods all increase the system consumption, and the additional consumption also uniformly increases the difficulty of maintaining the entire system, reducing the stability of the system.

[0004] On the other hand, the FTP / SFTP tools currently used for multiple file transmission still use a single-threaded queue method, and when there are many files to be transmitted, single-threading will reduce the efficiency of the entire file transmission. In addition, the currently widely used FTP / SFTP connection file synchronization method needs to start from the beginning when an exception occurs, such as when a file does not exist, the file path is incorrect, or the network is abnormal, and the transmitted part of the file cannot be resumed, and the already transmitted file also needs to be transmitted again, which consumes a lot of synchronization time.

[0005] Therefore, how to provide an efficient, easy-to-maintain and breakpoint-resuming data transmission method has become a technical problem to be solved. SUMMARY

[0006] Therefore, in order to overcome the shortcomings of the prior art, the present application aims to provide a non-structured file transmission method based on Flink.

[0007] The application provides a non-structured file transmission method based on Flink, comprising:

[0008] Step S1: reading source file configuration information, obtaining file quantity and file synchronization execution thread quantity;

[0009] Step S2: distributing file transmission tasks according to the obtained file quantity and file synchronization execution thread quantity;

[0010] Step S3: checking target files, creating source file server connection for the target files passing the check, and querying corresponding source files;

[0011] Step S4: creating target file server connection for the source files obtained by the query, and writing the target files into corresponding target file paths.

[0012] As a preferred embodiment of the application, in the non-structured file transmission method based on Flink, step S1 comprises reading file list attribute and execution thread quantity attribute of the source file configuration information, obtaining file quantity according to the read file list attribute, and obtaining file synchronization execution thread quantity according to the read execution thread quantity attribute.

[0013] As a preferred embodiment of the application, in the non-structured file transmission method based on Flink, step S1 further comprises ending file transmission when the read source file configuration information is illegal.

[0014] As a preferred embodiment of the application, in the non-structured file transmission method based on Flink, step S2 comprises distributing the same transmission task to each file synchronization execution thread, and distributing all the remaining transmission tasks to the remaining file execution threads.

[0015] As a preferred embodiment of the application, in the non-structured file transmission method based on Flink, in step S2, the quantity of the transmission task distributed to each file synchronization execution thread is obtained in the following manner: obtaining the ratio of the file quantity to the file synchronization execution thread quantity, rounding down the ratio, and taking the rounded-down value as the quantity of the transmission task distributed to each file synchronization execution thread.

[0016] As a preferred embodiment of the application, in the non-structured file transmission method based on Flink, step S3 comprises:

[0017] checking whether the configuration information of the target file and the source file configuration information have the same file id configuration, determining that the target file passes the check when the same file id configuration exists, and ending the file transmission corresponding to the target file when the same file id configuration does not exist;

[0018] create a source file server connection for the target file to be verified, when the source file server connection is successfully created, query whether the corresponding source file exists on the source file server through the source file server connection, when the corresponding source file does not exist on the source file server, end the file transmission corresponding to the target file;

[0019] When the source file server connection creation fails, save the error information to the log file, and end the file transmission corresponding to the target file.

[0020] As a preferred embodiment of the present application, in the Flink-based unstructured file transmission method of the present application, the source file server connection in step S3 includes an FTP connection or an SFTP connection.

[0021] As a preferred embodiment of the present application, in the Flink-based unstructured file transmission method of the present application, step S4 includes:

[0022] create a target file server connection for the queried source file, when the target file server connection is successfully created, query the corresponding target file path according to the configuration information of the target file through the target file server connection;

[0023] When the target file path does not exist, create the target file path of the target file and create a target file temporary file, obtain the corresponding read file stream and write stream through the source file server connection in step S3, and write the read file stream and the write stream into the target file temporary file;

[0024] When the target file path exists, query whether the corresponding target file temporary file exists under the target file path, and the target file temporary file is named with the target file name and suffixed with.temp;

[0025] When the corresponding target file temporary file does not exist under the target file path, create a target file temporary file, obtain the corresponding read file stream and write stream through the source file server connection in step S3, and write the read file stream and the write stream into the target file temporary file;

[0026] When the corresponding target file temporary file exists under the target file path, obtain the cache size of the target file temporary file, and write the read file stream after skipping the corresponding cache size and the write file stream into the target file temporary file;

[0027] modify the file name of the target file temporary file to the file name of the target file, and when the target file already exists under the target file path, overwrite the existing target file;

[0028] The file id configuration of the target file written is saved into the global cache checkpoint, and until all the tasks allocated to the thread are executed, the checkpoint information of the task is cleared, and the task state is marked as completed.

[0029] As a preferred embodiment of the present application, the Flink-based unstructured file transmission method further comprises: when the file transmission occurs an exception, restarting the file transmission task, and reading the global cache checkpoint when allocating the transmission task, if the file id configuration of the transmission file already exists in the global cache checkpoint, skipping the transmission of the transmission file.

[0030] Finally, the present application also provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above-mentioned Flink-based unstructured file transmission method when executing the program.

[0031] The Flink-based unstructured file transmission method has the following beneficial effects:

[0032] 1. When users face unstructured data in the scenario of data synchronization, they can also use the Flink tool to complete it, reducing the additional system consumption of users for unstructured file synchronization scenarios, reducing the deployment difficulty, reducing the maintenance difficulty, and improving the system stability.

[0033] 2. Multi-thread transmission is supported, and the number of files to be transmitted by each thread is allocated based on the number of files and the number of thread executions, improving the transmission efficiency and reducing the time consumed by the entire synchronization compared with single-thread transmission.

[0034] 3. When system exceptions such as host downtime and network interruption occur during file transmission, the checkpoint mechanism can restore the transmission progress to the progress before the exception and continue the previous task. For the file being written, the file size that has been written is skipped to continue appending writing, thereby avoiding repeated transmission and repeated reading and writing of the file, reducing the additional reading and writing consumption of the disk, and also reducing the time consumed by the entire synchronization. BRIEF DESCRIPTION OF DRAWINGS

[0035] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiments. Obviously, the drawings in the following description are only some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor on the basis of these drawings.

[0036] Figure 1A flow chart of a Flink-based unstructured file transmission method according to an exemplary first embodiment of the present application.

[0037] Figure 2 A flow chart of a Flink-based unstructured file transmission method according to an exemplary third embodiment of the present application.

[0038] Figure 3 A structural schematic diagram of the device provided by the present application. DETAILED DESCRIPTION

[0039] The embodiments of the present application will be described in detail below with reference to the accompanying drawings.

[0040] It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict; and all other embodiments obtained by those skilled in the art based on the embodiments in the present disclosure without creative labor are within the scope of protection of the present disclosure.

[0041] It should be noted that various aspects of the embodiments described below are within the scope of the appended claims. As will be apparent, the aspects described herein can be implemented in various forms and the specific structures and / or functions described herein are merely illustrative. Based on the present disclosure, one of ordinary skill in the art will appreciate that one aspect described herein can be implemented independently of any other aspects and that two or more of these aspects can be combined in various ways. For example, an apparatus can be implemented or a method can be practiced using any number of the aspects set forth herein. In addition, such an apparatus can be implemented or such a method can be practiced using other structure and / or functionality in addition to or other than one or more of the aspects set forth herein.

[0042] Figure 1 A flow chart of a Flink-based unstructured file transmission method according to an exemplary first embodiment of the present application. As shown in Figure 1 the method of the present embodiment comprises:

[0043] Step S1: reading source file configuration information to obtain the number of files and the number of file synchronization execution threads;

[0044] Step S2: allocating file transmission tasks according to the obtained number of files and the number of file synchronization execution threads;

[0045] Step S3: verifying target files, creating a source file server connection for the target files that pass the verification, and querying the corresponding source files;

[0046] Step S4: creating a target file server connection for the source files obtained by the query, and writing the target files into the corresponding target file path.

[0047] In practical application, step S1 of the embodiment is implemented in the following manner:

[0048] The file list attribute and the execution thread number attribute of the source file configuration information are read, the number of files is obtained according to the read file list attribute, and the number of file synchronous execution threads is obtained according to the read execution thread number attribute. When the read source file configuration information is illegal, the file transmission is ended. Specifically, in the embodiment, the source file configuration information is shown in Table 1.

[0049] Table 1

[0050]

[0051] As shown in Table 1, the attribute type of the file list is an array, and the array length identifies the number of files. The attribute type of the execution thread number is an integer.

[0052] The second exemplary embodiment of the present application provides a non-structured file transmission method based on Flink, which is a preferred embodiment of the method shown in Figure 1 The second exemplary embodiment of the present application provides a non-structured file transmission method based on Flink, which is a preferred embodiment of the method shown in

[0053] The same transmission task is allocated to each file synchronous execution thread, and the remaining transmission task is allocated to the remaining file execution thread. Specifically, the number of transmission tasks allocated to each file synchronous execution thread is obtained in the following manner: the ratio of the number of files to the number of file synchronous execution threads is obtained, the ratio is rounded down, and the rounded value is taken as the number of transmission tasks allocated to each file synchronous execution thread.

[0054] In practical application, when the number of files is S and the number of file synchronous threads is N, the number of transmission tasks allocated to each synchronous execution thread is calculated according to the formula floor(S / N), and the remaining transmission task allocated to the remaining file execution thread is calculated according to S-floor(S / N).

[0055] Figure 2 A flowchart of a non-structured file transmission method based on Flink according to the third exemplary embodiment of the present application, which is a preferred embodiment of the method shown in Figure 1

[0056] As shown in Figure 2 The third exemplary embodiment of the present application provides a non-structured file transmission method based on Flink, which is a preferred embodiment of the method shown in

[0057] The configuration information of the target file is verified with the source file configuration information to determine whether there is the same file id configuration. When there is the same file id configuration, it is determined that the target file passes the verification, and when there is no same file id configuration, the file transmission corresponding to the target file is ended. ​

[0058] A source file server connection is created for the target file that has passed verification. When the source file server connection is successfully created, the source file server connection is used to query whether the corresponding source file exists on the source file server. If the corresponding source file does not exist on the source file server, the file transfer corresponding to the target file is terminated.

[0059] If the connection to the source file server fails to be established, the error information is saved to the log file, and the file transfer corresponding to the target file is terminated.

[0060] In practical applications, the source file server connection in step S3 of this embodiment includes either an FTP connection or an SFTP connection. When using an FTP connection, an FTP connection is created based on the address, port, username, password, and connection mode. When using an SFTP connection, it is first determined whether the configuration contains a private key file path. If so, an SFTP connection is created using the private key; otherwise, an SFTP connection is created based on the username and password.

[0061] like Figure 2 As shown, step S4 of the method in this embodiment is implemented in the following manner:

[0062] A target file server connection is created for the source file obtained from the query. Once the target file server connection is successfully created, the corresponding target file path is queried through the target file server connection based on the configuration information of the target file.

[0063] When the target file path does not exist, the target file path is created and a temporary target file is created. The corresponding read file stream and write stream are obtained through the source file server connection in step S3, and the read file stream and write stream are written into the temporary target file.

[0064] If the target file path exists, check if a corresponding temporary file for the target file exists under the target file path. The temporary file for the target file is named after the target file name and has .temp as its suffix.

[0065] If no corresponding temporary file for the target file does not exist under the target file path, a temporary file for the target file is created. The corresponding read file stream and write stream are obtained through the source file server connection in step S3, and the read file stream and write stream are written into the temporary file for the target file.

[0066] If a corresponding temporary file for the target file exists under the target file path, the cache size of the temporary file for the target file is obtained. The read file stream skips the corresponding cache size and writes it into the temporary file for the target file along with the write file stream, thereby avoiding the retransmission of the file after the file transmission is interrupted and realizing the function of resuming interrupted transmission.

[0067] The file name of the target file temporary file is modified as the file name of the target file, and the existing target file is overwritten when the target file exists in the target file path;

[0068] The file id configuration of the written target file is saved to the global cache checkpoint, and the checkpoint information of the task is cleared after the execution of all tasks allocated to the thread is completed, and the task state is marked as completed.

[0069] In the method of the embodiment, when the file transmission occurs an exception, the file transmission task is restarted, and the global cache checkpoint is read when the transmission task is allocated. If the file id configuration of the transmission file already exists in the global cache checkpoint, the transmission of the transmission file is skipped.

[0070] As shown in Figure 3 The application further provides a device, including a processor 110, a communication interface 120, a memory 130 for storing a processor executable computer program, and a communication bus 140. Wherein the processor 110, the communication interface 120 and the memory 130 complete the communication among each other through the communication bus 140. The processor 110 realizes the above-mentioned Flink-based unstructured file transmission method by running the executable computer program.

[0071] Wherein, the computer program in the memory 130 can be realized in the form of a software function unit and sold or used as an independent product, which can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application essentially or say the part that contributes to the prior art or part of the technical solutions can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a plurality of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the embodiments of the present application. The foregoing storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various program code storage media.

[0072] The system embodiments described above are merely illustrative, wherein the units illustrated as separate components can or can not be physically separated, and the components illustrated as units can or can not be physical units, i.e., can be located in one place, or can be distributed to multiple network units. Part or all of the modules can be selected based on actual needs to achieve the purposes of the embodiments. Those skilled in the art can understand and implement without creative labor.

[0073] Through the description of the above embodiments, those skilled in the art can clearly understand that the embodiments can be implemented by means of software and the necessary universal hardware platform, and of course, can also be implemented by hardware. Based on such understanding, the above technical solutions can be embodied in the form of a software product, which can be stored in a computer readable storage medium, such as a ROM / RAM, a magnetic disk, an optical disk, etc., and includes a plurality of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute the methods of the embodiments or some parts of the embodiments.

[0074] The above is merely a specific implementation of the present application, but the protection scope of the present application is not limited thereto, and any person skilled in the art can easily think of changes or replacements within the technical range disclosed by the present application, which should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A Flink-based unstructured file transmission method, characterized in that, The method comprises: Step S1: reading source file configuration information, obtaining file quantity and file synchronization execution thread number; Step S2: allocating file transmission tasks according to the obtained file quantity and file synchronization execution thread number; Step S3: checking target files, creating a source file server connection for a target file that passes the check, and querying a corresponding source file; Step S4: creating a target file server connection for the queried source file, and writing the target file into a corresponding target file path; Step S4 comprises: creating a target file server connection for the queried source file, and querying a corresponding target file path through the target file server connection according to the configuration information of the target file when the target file server connection is successfully created; when the target file path does not exist, creating a target file path of the target file and creating a target file temporary file, obtaining a corresponding read file stream and a write stream through the source file server connection in step S3, and writing the read file stream and the write stream into the target file temporary file; when the target file path exists, querying whether a corresponding target file temporary file exists under the target file path, the target file temporary file being named with a target file name and having a.temp suffix; when the corresponding target file temporary file does not exist under the target file path, creating a target file temporary file, and obtaining a corresponding read file stream and a write stream through the source file server connection in step S3, and writing the read file stream and the write stream into the target file temporary file; when the corresponding target file temporary file exists under the target file path, obtaining a cache size of the target file temporary file, and writing the read file stream and the write stream into the target file temporary file after skipping the corresponding cache size; modifying the file name of the target file temporary file to the file name of the target file, and covering the existing target file when the target file already exists under the target file path; saving the file id configuration of the written target file to the global cache checkpoint, and clearing the checkpoint information of the task and marking the task state as completed after all tasks allocated to the thread are executed. 2.The Flink-based unstructured file transfer method according to claim 1, wherein, Step S1 comprises: reading file list attributes and execution thread number attributes of the source file configuration information, obtaining the file quantity according to the read file list attributes, and obtaining the file synchronization execution thread number according to the read execution thread number attributes. 3.The Flink-based unstructured file transfer method according to claim 1, wherein, Step S1 further comprises: ending the file transmission when the read source file configuration information is illegal. 4.The Flink-based unstructured file transmission method according to claim 1, wherein, Step S2 comprises: allocating the same transmission task to each file synchronization execution thread, and allocating all remaining transmission tasks to remaining file execution threads. 5.The Flink-based unstructured file transfer method according to claim 4, wherein, In step S2, the number of transmission tasks allocated to each file synchronization execution thread is obtained in the following manner: obtaining the ratio of the file quantity to the file synchronization execution thread number, taking the ratio down, and taking the integer part of the ratio as the number of transmission tasks allocated to each file synchronization execution thread. 6.The Flink-based unstructured file transfer method according to claim 1, wherein, Step S3 comprises: checking whether the configuration information of the target file and the configuration information of the source file have the same file id configuration, when the same file id configuration exists, determining that the target file passes the check, and when the same file id configuration does not exist, ending the file transmission corresponding to the target file; creating a source file server connection for the target file that passes the check, when the source file server connection is successfully created, querying whether the corresponding source file exists on the source file server through the source file server connection, when the corresponding source file does not exist on the source file server, ending the file transmission corresponding to the target file; when the source file server connection fails to be created, saving error information to a log file, and ending the file transmission corresponding to the target file. 7.The Flink-based unstructured file transfer method according to claim 1, wherein, The source file server connection in step S3 includes an FTP connection or an SFTP connection. 8.The Flink-based unstructured file transfer method according to claim 1, wherein, The method further includes: when an exception occurs in the file transmission, restarting the file transmission task, and reading a global cache checkpoint when the transmission task is allocated, if the file id configuration of the transmission file already exists in the global cache checkpoint, skipping the transmission of the transmission file.

9. A computer device, comprising: The computer device includes a memory, a processor, and a computer program stored on the memory and executable on the processor, and the processor implements the steps of the method in any one of claims 1-8 when executing the program.

Images