Minio file transmission method and device based on DataX and electronic equipment

Abstract

The invention relates to a Minio file transmission method and device based on DataX, and aims to solve the problems that an existing file transmission method is low in transmission efficiency, insufficient in reliability and lack of standardized integration. The method comprises the following steps: configuring Minio as a source data source or a target data source of a data transmission task through a task configuration file of DataX, and setting related transmission parameters; a read-write plug-in suitable for Minio is developed and integrated into the DataX framework, and direct reading and writing of files in a Minio storage bucket are achieved; after the large file is fragmented, multi-thread transmission is carried out by utilizing a DataX concurrency mechanism, so that the file transmission efficiency is improved; the fragmentation state is recorded in the transmission process, breakpoint resume is supported, and the reliability is ensured; and after the transmission is completed, verifying the file integrity through a checksum algorithm. Standardized integration of the DataX and the Minio is achieved through the Minio data source plug-in, the use threshold of a user is reduced, in addition, on the basis of implementation of an open source tool, the deployment and maintenance cost is reduced, and the method is suitable for small and medium-sized enterprises and lightweight application scenes.

Description

Technical Field

[0001] This application belongs to the technical field of file transfer, and particularly relates to a Minio file transfer method, device, computer-readable storage medium, and electronic device based on DataX. Background Art

[0002] In the field of big data, file transfer and storage are the basis of data processing. Currently, common file transfer and storage technologies mainly include the following: FTP / SFTP transfer: FTP (File Transfer Protocol) and SFTP (Secure File Transfer Protocol) are traditional file transfer methods that use the client-server mode to achieve file upload and download. However, their transfer speed is slow, and they cannot make full use of bandwidth resources; they do not support the resume function; and their security highly depends on configuration, and there are security risks if the configuration is improper.

[0003] HDFS storage and transfer: The Hadoop Distributed File System (HDFS) is a commonly used storage solution for processing large-scale data. However, its deployment and maintenance costs are high, making it unaffordable for some small and medium-sized enterprises or lightweight application scenarios; and its support efficiency for small files is low, and its performance is poor when dealing with a large number of small files.

[0004] Object storage technology: Object storage technology (such as Amazon S3, Minio) has gradually become the mainstream storage solution due to its advantages such as high scalability, low cost, and high availability. As an open-source S3-compatible object storage system, Minio is widely used in private cloud and hybrid cloud environments. However, Minio itself lacks an efficient data transfer tool and usually needs to rely on third-party tools or custom scripts to achieve file transfer.

[0005] DataX data transfer tool: DataX is an efficient data synchronization tool open-sourced by Alibaba, which can support data transfer between multiple data sources. However, DataX does not natively support Minio as a data source or target storage, and users need to perform additional development and adaptation work.

[0006] In summary, the existing file transfer and storage technologies mainly have the following deficiencies: (1) Low transfer efficiency: Traditional transfer methods (such as FTP / SFTP) have a slow transfer speed and low transfer efficiency in big data scenarios, cannot make full use of bandwidth resources, and are difficult to meet the demand for rapid transfer of large-scale data.

[0007] (2) Lack of standardized integration: As an object storage system, Minio lacks standardized integration with mainstream data transfer tools (such as DataX), resulting in users having to develop adaptation code by themselves when using it, increasing the development cost and workload, and reducing the overall usability of the system.

[0008] (3) Insufficient reliability: During the transmission process of existing file transfer tools, transmission interruptions may occur due to factors such as network fluctuations and system failures. Moreover, most of these tools lack a perfect resume mechanism. Once the transmission is interrupted, it needs to start over, further reducing the transmission efficiency and reliability.

[0009] (4) High operation and maintenance costs: Storage solutions such as HDFS have a relatively complex deployment and maintenance process, requiring professional technical personnel to operate and maintain. Therefore, they are not suitable for small and medium-sized enterprises or lightweight application scenarios, restricting their application in a wider range. Summary of the Invention

[0010] To address the above problems, this application proposes a new Minio file transfer method and device based on DataX, aiming to solve the problems of low transmission efficiency, complex integration, and insufficient reliability in existing file transfer methods, and achieve efficient, reliable, and low-cost file transfer and storage.

[0011] This application mainly adopts the following technical strategies: (1) Standardized integration of Minio and DataX: By developing a Minio data source plugin, seamless integration of the Minio object storage system and the DataX data transfer framework is achieved.

[0012] Advantages: Users do not need to develop additional adaptation code, reducing the usage threshold; supporting Minio as the source data source and target data source of DataX, expanding the application scenarios of DataX.

[0013] (2) Efficient sharded upload mechanism: Combining the sharded upload function of Minio and the multi-threaded concurrency mechanism of DataX, efficient transmission of large files is achieved.

[0014] Advantages: Splitting large files into multiple shards and uploading them in parallel significantly improves the transmission speed; supporting dynamic adjustment of the shard size and concurrency number to adapt to different network environments and file sizes.

[0015] (3) Reliable sharded resume: Introducing a resume mechanism during the sharded upload process, recording the upload status of each shard, and supporting resuming upload from the interruption point.

[0016] Advantages: Avoid transmission failures caused by network fluctuations or system failures, and improve the reliability of transmission; For incomplete shards, only the remaining parts need to be uploaded, saving bandwidth and time.

[0017] (4) Data integrity verification mechanism: Calculate the checksum (such as MD5, SHA256, etc.) of the source file and the target file before and after file transmission to ensure the integrity and accuracy of data transmission.

[0018] Advantages: Avoid data corruption or loss through checksum comparison; When the verification fails, trigger the retry mechanism to further improve the reliability of transmission.

[0019] (5) Flexible configuration and management: Dynamically adjust transmission parameters (such as shard size, concurrency, verification algorithm, etc.) through the configuration file, and provide flexible task management functions.

[0020] Advantages: Users can optimize transmission performance according to their needs; Support the persistent storage and management of task status, facilitating task monitoring and troubleshooting.

[0021] (6) Low cost and easy scalability: Based on open-source DataX and Minio, reduce deployment and maintenance costs, and at the same time support expansion to other object storage systems (such as Amazon S3, Alibaba Cloud OSS, etc.).

[0022] Advantages: Suitable for small and medium-sized enterprises and lightweight application scenarios; Through standardized interfaces, it is easy to expand and integrate with other storage systems.

[0023] (7) Multi-threaded concurrent transmission: Combine the multi-threaded concurrent mechanism of DataX to achieve multi-task parallel transmission and make full use of system resources.

[0024] Advantages: Significantly improve transmission efficiency, especially suitable for large-scale data transmission scenarios; Manage concurrent tasks through the thread pool to optimize resource utilization.

[0025] Generally speaking, this application provides a Minio file transmission method based on DataX, which mainly includes the following steps: Data transmission task configuration: Through the task configuration file of DataX, configure Minio as the source data source or target data source of the data transmission task, and set relevant transmission parameters, including but not limited to file path, shard size, concurrency, etc.

[0026] Configure Minio data source read and write plugins: Develop and integrate read and write plugins suitable for Minio data sources into the DataX framework, so as to achieve direct reading and writing operations of files in Minio buckets by DataX.

[0027] File Sharding and Concurrent Transmission: Split the large file to be transmitted into multiple smaller shards, and use the concurrent mechanism of DataX to achieve multi-threaded transmission, thereby improving the file transmission efficiency.

[0028] Resume Transmission Mechanism: During the file transmission process, record the transmission status of each shard in real time; if the transmission is interrupted due to network fluctuations, system failures, etc., it can continue to transmit from the breakpoint where the last interruption occurred when resuming the transmission, thus ensuring the reliability of the transmission process.

[0029] Verification of Transmission Results: After the file transmission is completed, verify the integrity of the transmitted file through the Checksum algorithm to ensure the consistency and accuracy of the data.

[0030] Through the above methods, this application solves the problems of low file transmission efficiency, insufficient reliability, and lack of standardized integration in the prior art, and provides an efficient, reliable, and easy-to-integrate solution for file transmission based on DataX and Minio.

[0031] Specifically, this application provides the following technical solutions: The first aspect of this application provides a Minio file transmission method based on DataX, and the method includes: Configure Minio as the source data source or target data source of the data transmission task through the task configuration file of DataX, and set relevant transmission parameters; Develop and integrate read and write plugins suitable for Minio data sources into the DataX framework, so that DataX can directly read and write files in Minio buckets; Split the large file to be transmitted into multiple shards, and use the concurrent mechanism of DataX to achieve multi-threaded transmission to improve the transmission efficiency; During the file transmission process, record the transmission status of each shard in real time. If the transmission is interrupted, continue the transmission from the breakpoint where the last interruption occurred when resuming the transmission; After the file transmission is completed, verify the integrity of the transmitted file through the Checksum algorithm to ensure the consistency and accuracy of the data.

[0032] Optionally, in the method of this application, the transmission parameters include but are not limited to: file path, shard size, and concurrency number.

[0033] Furthermore, in the method of this application, the read and write plugins include a read plugin and a write plugin, where: The read plugin distributes multiple files to multiple subtasks respectively by processing files concurrently, and realizes multi-threaded transmission through the concurrent mechanism of DataX; The write plugin supports the segmented upload operation of large files, including segment division, initialization of segmented upload, segmented upload, segmented resumption of interrupted transfer, and merging of segments.

[0034] Furthermore, in the method of the present application, the segmented upload operation includes the following steps: (1) Split a single large file into multiple segments, and the size of each segment is dynamically adjusted according to the network bandwidth and file size; (2) Use the MinioClient client to initialize the segmented upload task and obtain a unique uploadId; (3) Upload each segment in a multi-threaded manner and record the upload status of each segment; (4) After the transmission is interrupted, detect the unfinished segments and continue the upload from the unfinished segments; (5) After all segments are uploaded, pass all segment information to the Minio server through the MinioClient client to complete the segment merging.

[0035] Furthermore, in the method of the present application, the checksum algorithm is the MD5 algorithm or the SHA256 algorithm.

[0036] Furthermore, the method of the present application further includes: during the file transmission process, record the transmission status of each segment in real time, and record the transmission status information of each segment through a distributed cache system, where the transmission status information includes segment number, upload status, and the number of bytes already uploaded.

[0037] Furthermore, the method of the present application further includes an exception handling mechanism. After the file transmission is completed, perform integrity verification on the transmitted file through the checksum algorithm. If the verification fails, mark the transmission task as failed and trigger a retry mechanism.

[0038] The second aspect of the present application provides a Minio file transmission device based on DataX, and the device includes: A task configuration module, used to configure Minio as the source data source or target data source of the data transmission task through the task configuration file of DataX, and set relevant transmission parameters; A read / write plugin module, used to develop and integrate a read / write plugin suitable for the Minio data source into the DataX framework, so that DataX can directly read and write files in the Minio bucket; A file division module, used to split the large file to be transmitted into multiple segments and use the concurrency mechanism of DataX to achieve multi-threaded transmission; A breakpoint resumption module, used to record the transmission status of each segment in real time during the file transmission process. If the transmission is interrupted, continue the transmission from the breakpoint where the last interruption occurred when resuming the transmission; A verification and comparison module, which is used to perform integrity verification on the transmitted file through a checksum algorithm after the file transmission is completed, so as to ensure the consistency and accuracy of the data.

[0039] When the device runs, it implements the steps of the aforementioned Minio file transfer method based on DataX.

[0040] The third aspect of this application provides an electronic device, including: a memory and a processor; Memory: used to store computer programs; Processor: used to execute the computer program to implement the steps of the aforementioned Minio file transfer method based on DataX.

[0041] The fourth aspect of this application provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, it implements the steps of the aforementioned Minio file transfer method based on DataX.

[0042] In summary, the Minio file transfer method based on DataX proposed in this application has the following advantages: (1) Efficient transmission: Through the file sharding and concurrent transmission mechanisms, it can make full use of network bandwidth resources, significantly improve the file transmission efficiency, and meet the fast transmission requirements in big data scenarios.

[0043] (2) Standardized integration: By using the Minio data source plugin, the standardized integration of DataX and Minio is achieved, reducing the user's usage threshold and improving the usability and compatibility of the system.

[0044] (3) High reliability: Through the breakpoint resumption mechanism and the transmission result verification function, the reliability and integrity of data transmission are ensured, and it can effectively cope with emergencies such as network fluctuations or system failures.

[0045] (4) Low cost: It is implemented based on open-source tools (DataX and Minio), reducing the system deployment and maintenance costs, and is suitable for small and medium-sized enterprises and lightweight application scenarios.

[0046] (5) Easy to expand: This method and device have good scalability and can be extended to other object storage systems (such as Amazon S3, Alibaba Cloud OSS, etc.), with broad application prospects.

[0047] Other features and advantages of the present application will be described in detail in the following specification, or can be understood by implementing the relevant technical solutions of the present application. The objectives and other advantages of the present application can be achieved by the technical features and means clearly pointed out in the specification, claims, and drawings, and obtained through the implementation process of these technical contents. Brief Description of the Drawings

[0048] To more clearly illustrate the technical solutions of the embodiments of the present application, the drawings involved in the description of the embodiments will be briefly introduced below. It should be noted that the drawings only show some embodiments of the present application. For those skilled in the art, without creative efforts, other relevant drawings can be derived based on these drawings.

[0049] Figure 1 It is the overall design architecture diagram of the solution of the present application.

[0050] Figure 2 It is the overall implementation flowchart of the Minio file transfer method based on DataX of the present application.

[0051] Figure 3 It is a schematic diagram of multi-threaded shard uploading in the write plugin of the solution of the present application.

[0052] Figure 4 It is the composition structure diagram of the Minio file transfer device based on DataX of the present application.

[0053] Figure 5 It is the structure schematic diagram of the electronic device provided by the embodiment of the present application. Detailed Embodiments

[0054] To make the objectives, technical solutions, and advantages of the embodiments of the present application clearer and more understandable, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. It should be clear that the described embodiments are only some embodiments of the present application, rather than all embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments in the present application without creative efforts belong to the protection scope of the present application.

[0055] In this text, the term "including" and any form of its variation (such as "including with", "including") are open-ended expressions and should be understood as "including but not limited to", that is, the listed content is not an exhaustive list and may also include other content not explicitly mentioned. The term "based on" should be understood as "at least partially based on", that is, the referred basis or condition may not be the only factor and may also involve other relevant factors. The term "an embodiment" should be understood as "at least one embodiment", that is, the described embodiment is not the only possible implementation and there may be other similar embodiments.

[0056] In this application, when the terms "a" and "multiple" are used to modify related elements or features, their expressions are illustrative rather than restrictive. Unless otherwise clearly stated in the context, "a" should be understood as "at least one", and "multiple" should be understood as "at least two". Those skilled in the art should make a reasonable interpretation of these terms according to the semantic and logical relationships in the context to ensure that they cover the possibility of "one or more".

[0057] Figure 1 and Figure 2 The overall implementation process of the Minio file transfer method based on DataX provided by this application is shown as follows, including the following steps: S1. Configure Minio as the source data source or target data source of the data transfer task through the task configuration file of DataX, and set relevant transfer parameters; S2. Develop and integrate read and write plugins suitable for the Minio data source into the DataX framework so that DataX can directly read and write files in the Minio bucket; S3. Split the large file to be transferred into multiple shards, and use the concurrent mechanism of DataX to achieve multi-threaded transfer to improve the transfer efficiency; S4. During the file transfer process, record the transfer status of each shard in real time. If the transfer is interrupted, resume the transfer from the breakpoint where the transfer was interrupted last time; S5. After the file transfer is completed, perform integrity verification on the transferred file through the checksum algorithm to ensure the consistency and accuracy of the data.

[0058] To more clearly elaborate on the technical solution of this application, the following will be further described through embodiments in specific scenarios.

[0059] The Minio file transfer device based on DataX provided by this application includes the following functional modules: (I) Task configuration module The task configuration module is one of the core components of this application, responsible for defining and configuring various parameters of the data transmission task to ensure that the data transmission process can be executed according to the predefined rules and conditions.

[0060] The main functions of the task configuration module include: (1)Define data sources and destinations: Clearly define the sources and destinations of data transmission, that is, from which Minio bucket to read data and then write the data to which Minio bucket.

[0061] (2)Set transmission parameters: Configure various parameters related to transmission, such as file path, shard size, concurrency number, etc.

[0062] (3)Generate a configuration file: Generate a task configuration file in JSON format recognizable by DataX from the configuration parameters for the DataX engine to execute.

[0063] The task configuration module needs to configure the following key parameters: (1)Data source configuration (Reader) name: The name of the data source plugin, fixed as minioreader.

[0064] parameter: The specific parameters of the data source, including: endpoint: The address of the Minio server, such as http: / / minio.source.com.

[0065] accessKey: The access key of Minio.

[0066] secretKey: The private key of Minio.

[0067] bucket: The name of the bucket where the source data is located.

[0068] object: The path of the source file, such as path / to / source / file.

[0069] splitSize: The file shard size (unit: byte), such as 5242880 (5MB).

[0070] concurrency: The number of threads for concurrent transmission, such as 4.

[0071] (2)Data destination configuration (Writer) name: The name of the data destination plugin, fixed as miniowriter.

[0072] parameter: The specific parameters of the data destination, including: endpoint: The address of the Minio server, e.g., http: / / minio.target.com.

[0073] accessKey: The access key of Minio.

[0074] secretKey: The private key of Minio.

[0075] bucket: The name of the bucket where the target data is located.

[0076] object: The path of the target file, e.g., path / to / target / file.

[0077] (3)Transfer speed configuration (Setting) speed: The transfer speed control parameter, including: channel: The number of concurrent channels, e.g., 3.

[0078] (2)Read and write plugin module The read and write plugin module is the core component for integrating DataX with the Minio object storage system, responsible for reading data from Minio and writing data to Minio.

[0079] The main functions of the read and write plugin module include: (1)Configuration management: Parse and manage Minio connection configurations (such as endpoint, accessKey, secretKey, etc.).

[0080] (2)File reading: Read files from the Minio bucket and pass the data to the DataX framework.

[0081] (3)File writing: Write the data passed by the DataX framework to the Minio bucket.

[0082] (4)Multipart upload: Support multipart upload of large files to improve transfer efficiency and reliability.

[0083] (5)Resume interrupted transfer: Used to record the transfer status and implement resume of interrupted file transfer.

[0084] The read and write plugin module includes: 1. Read plugin To improve the file transfer efficiency, a concurrent processing scheme for files is adopted in the read plugin. In this embodiment, relying on the task splitting mechanism of DataX, in the split method of the Job, DataX will split the task into multiple subtasks according to the configured number of concurrent channels (channel), and each subtask is executed by an independent thread.

[0085] In this embodiment, a scheme of splitting by file unit is adopted, and multiple files are respectively divided into multiple subtasks (the number of subtasks is the same as the number of configured channels).

[0086] The specific allocation calculation method is as follows: number the files to be read at the source end, and then take the modulus of the number of each file by the configured concurrency number (channel), and the obtained result is the subtask number allocated to each file.

[0087] Assume that the concurrency number is 3 and there are 7 files at the source end. After numbering them respectively, according to the scheme of this application, the subtask file allocation results calculated are shown in Table 1 below.

[0088] Table 1 Subtask File Allocation Result Table in the Read Plugin

[0089] Through the above division, the number of files processed by each subtask is reduced. Multiple subtasks are processed in parallel, and the multi-threaded transmission is realized by using the concurrency mechanism of DataX, which can significantly improve the file transmission efficiency.

[0090] After the file set to be processed by each subtask is allocated, the specific file reading process needs to be carried out. The implementation steps for the subtask to read files from Minio are as follows: (1) Use the MinioClient client to read the files in the specified bucket in Minio into the memory in the form of an InputStream input stream; (2) Write the input stream into a byte array in the form of a byte array and set it into the Record object of DataX; (3) Use RecordSender to send the Record object to the write plugin.

[0091] 2. Write Plugin In the write plugin, the same number of subtasks as in the read plugin will also be initialized to process the allocated files.

[0092] In the write plugin, segmented uploading is the core function to achieve efficient transmission of large files, and mainly includes the following steps: segmented division, initialization of segmented uploading, segmented uploading, segmented breakpoint resumption, and merging of segments.

[0093] (1) Segmented Division To reduce the amount of data uploaded at one time and improve the upload speed, in this solution, a scheme of splitting a single large file into multiple small chunks and uploading them to the Minio server separately is adopted. The shard size is the number of bytes per shard, usually set between 5MB and 1GB. Here, an example with a shard size of 5MB is used for illustration.

[0094] The total number of shards totalParts = the number of bytes of a single file / 5 * 1024 * 1024 + 1 The shard size can be dynamically adjusted according to the network bandwidth and file size.

[0095] (2)Initialize multipart upload Initializing multipart upload is to apply for a multipart upload task to the Minio server and obtain a unique uploadId.

[0096] In this embodiment, the initiateMultipartUpload method of the MinioClient client is used to initialize a multipart upload task. This method will return an upload ID, which is used to identify the upload tasks of different files.

[0097] String uploadId = minioClient.initiateMultipartUpload(BUCKET_NAME,objectName); Among them, BUCKET_NAME is the bucket name in Minio, and objectName represents the file name to be uploaded.

[0098] (3)Multipart upload As Figure 3 shown, in this solution, a thread pool is adopted, and multiple threads are used to perform the Minio multipart upload operation. This can avoid the performance bottleneck of single-threaded processing and can significantly improve the upload efficiency.

[0099] The implementation steps of multipart upload are as follows: 1) Initialize the thread pool. The available number of threads is based on the number of threads configured by the concurrency parameter; 2) The write plugin obtains the data transferred from the source end from the RecordReceiver and converts it into a byte array for each shard; 3) Convert the byte array of each shard into a ByteArrayInputStream byte input stream for each shard and record the sequence number of each shard; 4) Use multiple threads to upload each shard through the uploadPart method of the MinioClient client in each thread and return the block information.

[0100] Here, multi-threading is used to execute the operation of uploading each shard to Minio in parallel, making full use of network bandwidth resources.

[0101] (4) Resuming fragmented uploads The main process of fragment breakpoint resuming includes: Upload status record: During the upload process, the upload status of each segment is recorded (such as whether it is completed, the number of bytes uploaded, etc.).

[0102] Interruption detection: After a task is interrupted, detect which shards have not been uploaded.

[0103] Resume upload: Continue uploading from the unfinished segment.

[0104] The detailed implementation steps are: 1) Upload status record The upload status of each shard needs to be persistently stored. Considering the deployment in a distributed environment, in this application, Redis (Remote Dictionary Server, an open source memory data structure storage system that can be used as a database, cache, and message middleware) is used to cache the transmission status information of each shard, including: shard sequence number, shard upload status (such as not started, in progress, completed), number of bytes uploaded, etc.

[0105] 2) Interrupt detection When the file transfer task is resumed, read the upload status record to detect the incomplete shards.

[0106] Unfinished segments need to be uploaded again.

[0107] 3) Resume upload For the uploaded shards, you need to skip uploading them when resuming the upload to avoid repeated uploading of files.

[0108] For the incomplete uploaded segments, you need to continue uploading from the number of bytes that have been uploaded, so that you can accurately resume the upload from the breakpoint.

[0109] (5) Merge shards After all the fragments are uploaded, you need to pass all the fragment information to the Minio server through the completeMultipartUpload method of the MinioClient client, and notify the Minio server to merge all the fragments of the specified file to obtain a complete file.

[0110] (III) Verification and comparison module The verification and comparison module is an important component in the Minio file transfer solution, which is used to ensure the integrity and accuracy of data transmission. Through the verification mechanism, the consistency of the source and target files can be verified after the file transfer is completed to avoid data damage or loss.

[0111] The main functions of the verification and comparison module include: (1) Calculate the checksum: Before and after the file transfer, calculate the checksum (such as MD5, SHA256, etc.) of the source file and the target file respectively.

[0112] (2) Verification and comparison: Compare the checksums of the source file and the target file to ensure that they are consistent.

[0113] (3) Exception handling: If the verification fails, the transfer task is marked as failed and the retry mechanism is triggered.

[0114] The verification and comparison process includes: (1) Verification algorithm Commonly used verification algorithms include: MD5: Generates a 128-bit hash value with fast calculation speed but low security.

[0115] SHA256: Generates a 256-bit hash value with high security, suitable for scenarios with high requirements for data integrity.

[0116] The user can select a suitable verification algorithm according to the requirements. In this embodiment, MD5 is selected as the verification algorithm.

[0117] (2) Verification process Calculate source file checksum before transfer: Calculate and save the checksum of the source file before transferring the file.

[0118] Calculate the checksum of the destination file after transfer: Calculate the checksum of the destination file after the file transfer is completed.

[0119] Verification and comparison: Compare the checksums of the source file and the target file. If they are consistent, the transfer is marked as successful, otherwise it is marked as failed.

[0120] (3) Exception handling If the verification fails, an error log is recorded and a retry mechanism is triggered.

[0121] The number of retries and strategies can be configured as required.

[0122] Figure 4 The figure shows a Minio file transmission device based on DataX proposed in this application, including: The task configuration module is used to configure Minio as the source data source or target data source of the data transmission task through the DataX task configuration file, and set the relevant transmission parameters; The read-write plugin module is used to develop and integrate read-write plugins for Minio data sources into the DataX framework, enabling DataX to directly read and write files in the Minio bucket; The file partitioning module is used to split the large file to be transmitted into multiple fragments and use the concurrency mechanism of DataX to achieve multi-threaded transmission; The breakpoint resume module is used to record the transmission status of each fragment in real time during the file transfer process. If the transmission is interrupted, the transmission will be resumed from the breakpoint of the last interruption when the transmission is resumed; The verification and comparison module is used to verify the integrity of the transmitted files through a checksum algorithm after the file transfer is completed to ensure the consistency and accuracy of the data.

[0123] When the above device is running, the steps of the Minio file transfer method based on DataX disclosed in this application are implemented.

[0124] The flowcharts and block diagrams in the accompanying drawings illustrate possible implementations of the apparatus, methods, and computer program products according to various embodiments of the present application, including architecture, functions, and operations. In these figures, each box may represent a module, a program segment, or a portion of a code, which contains one or more executable instructions for implementing a specified logical function. It should be noted that each box in the block diagram and / or flowchart, as well as the combination of these boxes, can be implemented using a dedicated hardware-based system to implement the specified function or operation, or can be implemented by a combination of dedicated hardware and computer instructions.

[0125] like Figure 5 As shown, the embodiment of the present application also discloses an electronic device, including: a processor 310, a communication interface 320, a memory 330 for storing a processor executable computer program, and a communication bus 340. The processor 310, the communication interface 320, and the memory 330 communicate with each other through the communication bus 340. The processor 310 runs the executable computer program to implement the steps of the above-mentioned Minio file transfer method based on DataX.

[0126] It is understandable that, in addition to the memory and the processor, the electronic device may also include an input device (such as a keyboard), an output device (such as a display) and other communication modules. These input devices, output devices and other communication modules communicate with the processor through an I / O interface (i.e., an input / output interface).

[0127] The operation of the present application can be implemented by writing computer program codes using one or more programming languages ​​or a combination thereof. The programming languages ​​include but are not limited to the following types: Object-oriented programming languages, such as Java, Smalltalk, C++, etc.; A conventional procedural programming language, such as "C" or a similar programming language.

[0128] The execution methods of program code include but are not limited to: Executes entirely on the user's computer; Partial execution on the user's computer and part execution on a remote computer; Implemented as a standalone package; Executes entirely on the remote computer or server.

[0129] In scenarios involving remote computers, the remote computer can be connected to the user's computer through any type of network, including but not limited to a local area network (LAN) or a wide area network (WAN). In addition, the remote computer can also be connected to an external computer through an Internet service provider, such as using the Internet.

[0130] Furthermore, the present application also discloses a computer-readable storage medium. When the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device can execute the various steps of the DataX-based Minio file transfer method disclosed in the present application.

[0131] In the context of this application, computer-readable storage media refers to tangible media that can store computer program code and related data. Specific examples include, but are not limited to, the following: (1) Portable computer disk: A removable magnetic storage medium such as a floppy disk.

[0132] (2) Hard disk: includes fixed storage devices such as mechanical hard disks and solid-state hard disks.

[0133] (3) Random Access Memory (RAM): Volatile storage medium used for temporary storage of data and program code.

[0134] (4) Read-only memory (ROM): A non-volatile storage medium used to store fixed programs and data.

[0135] (5) Erasable Programmable Read-Only Memory (EPROM) or Flash Memory: A non-volatile storage medium that supports multiple erasing and programming.

[0136] (6) Fiber optic storage device: storage medium based on fiber optic technology.

[0137] (7) Portable Compact Disc Read Only Memory (CD-ROM): A read-only medium that stores data in the form of an optical disc.

[0138] (8) Optical storage devices: storage media based on optical principles, such as DVDs and Blu-ray discs.

[0139] (9) Magnetic storage devices: storage media based on magnetic principles, such as magnetic tapes and disks.

[0140] (10) Any suitable combination of the above: for example, combining multiple storage media to meet different storage requirements.

[0141] These computer-readable storage media can be used to store the program code and related data described in this application to support the operation of the program and the persistent storage of data.

[0142] In particular, according to an embodiment of the present application, the process described in the flowchart can be implemented as a computer software program. For example, an embodiment of the present application relates to a computer program product, which includes a computer program carried on a non-transitory computer-readable medium. The computer program includes program code for executing the DataX-based Minio file transfer method disclosed in the present application. When the computer program is executed by a processing device, the above functions defined in the embodiments of the present application can be implemented.

[0143] Although the above discussion contains some specific implementation details, these details should not be interpreted as limiting the scope of this application. The above description is only a preferred embodiment of the present application and an explanation of the technical principles used. Those skilled in the art should understand that the disclosure scope involved in this application is not limited to the technical solutions formed by the specific combination of the above technical features. At the same time, this application should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above public concept.

[0144] Those skilled in the art should also understand that they can modify the technical solutions described in the above embodiments without departing from the spirit and scope of the technical solutions of the embodiments of the present application, or replace some of the technical features therein by equivalents. These modifications or replacements will not cause the essence of the corresponding technical solutions to deviate from the core spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A Minio file transfer method based on DataX, characterized in that: The method comprises: Use the DataX task configuration file to configure Minio as the source data source or target data source for the data transmission task, and set the relevant transmission parameters; Develop and integrate read and write plugins for Minio data sources into the DataX framework, enabling DataX to directly read and write files in Minio buckets; Split the large file to be transferred into multiple shards and use DataX's concurrency mechanism to implement multi-threaded transmission; During the file transfer process, the transfer status of each segment is recorded in real time. If the transfer is interrupted, the transfer will be resumed from the last interruption point when the transfer is resumed. After the file transfer is completed, the integrity of the transferred file is verified through a checksum algorithm.

2. The method according to claim 1, characterized in that: The transmission parameters include: file path, slice size and concurrency number.

3. The method according to claim 1, characterized in that The read-write plug-in includes a read plug-in and a write plug-in, wherein: The read plug-in processes files concurrently, assigns multiple files to multiple subtasks, and implements multi-threaded transmission through DataX's concurrent mechanism; The writing plug-in supports multi-part upload operations for large files, including multi-part division, initial multi-part upload, multi-part upload, multi-part breakpoint resume and multi-part merging.

4. The method according to claim 3, characterized in that The segment upload operation includes the following steps: (1) Split a single large file into multiple fragments, and dynamically adjust the size of each fragment based on network bandwidth and file size; (2) Use the MinioClient client to initialize the shard upload task and obtain a unique uploadId; (3) Use multi-threading to upload each shard and record the upload status of each shard; (4) After the transmission is interrupted, detect the unfinished segments and continue uploading from the unfinished segments; (5) After all shards are uploaded, all shard information is passed to the Minio server through the MinioClient client to complete the shard merging.

5. The method according to claim 1, characterized in that The checksum algorithm is the MD5 algorithm or the SHA256 algorithm.

6. The method according to claim 1, characterized in that The method also includes: during the file transmission process, recording the transmission status of each shard in real time, and recording the transmission status information of each shard through a distributed cache system, wherein the transmission status information includes the shard sequence number, upload status and the number of bytes uploaded.

7. The method according to claim 1, characterized in that The method also includes an exception handling mechanism. When the file transfer is completed, the integrity of the transferred file is verified by a checksum algorithm. If the check fails, the transfer task is marked as failed and a retry mechanism is triggered.

8. A Minio file transfer device based on DataX, characterized in that: The device comprises: The task configuration module is used to configure Minio as the source data source or target data source of the data transmission task through the DataX task configuration file, and set the relevant transmission parameters; The read-write plugin module is used to develop and integrate read-write plugins for Minio data sources into the DataX framework, enabling DataX to directly read and write files in the Minio bucket; The file partitioning module is used to split the large file to be transmitted into multiple fragments and use the concurrency mechanism of DataX to achieve multi-threaded transmission; The breakpoint resume module is used to record the transmission status of each fragment in real time during the file transfer process. If the transmission is interrupted, the transmission will be resumed from the breakpoint of the last interruption when the transmission is resumed; The verification and comparison module is used to verify the integrity of the transmitted files through a checksum algorithm after the file transfer is completed to ensure the consistency and accuracy of the data.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the Minio file transfer method based on DataX as described in any one of claims 1 to 7 are implemented.

10. An electronic device, characterized in that: include: Memory and processor; Memory: used to store computer programs; Processor: used to execute the computer program to implement the steps of the Minio file transfer method based on DataX as described in any one of claims 1-7.

Images