A datax-based s3 data synchronization method and device

Abstract

The application discloses a kind of S3 data synchronization method and device based on DataX.This application constructs native s3reader and s3writer plug-in under DataX framework, directly calls S3 SDK to realize streaming read-write.The reading end utilizes GetObject input stream to analyze in real time as DataX Record;The writing end adopts double-buffering mechanism to accumulate data in memory, and asynchronously triggers Multipart Upload fragment upload, realizes full-link data zero disk landing.In addition, the application proposes a dynamic task splitting algorithm based on file size balancing to avoid data skew;And through Multipart Upload mechanism, the atomicity and consistency of data writing are guaranteed.This application has the advantages of no Hadoop dependency, zero disk I / O, dynamic load balancing and high reliability, which significantly improves the synchronization efficiency between S3 and heterogeneous data sources.

Description

Technical Field

[0001] This application relates to the field of big data and cloud computing data synchronization technology, specifically to an S3 data synchronization method, apparatus, computer-readable storage medium, and electronic device based on DataX. Background Technology

[0002] DataX is an open-source offline synchronization tool for heterogeneous data sources from Alibaba. It adopts a "framework + plugin" architecture design, abstracting the data source reader and writer into standardized plugin interfaces, and supports stable and efficient data synchronization between various data sources such as relational databases (such as MySQL and Oracle), HDFS, HBase, and FTP.

[0003] With the widespread adoption of cloud computing, Amazon S3 and its compatible object storage systems (such as MinIO, Alibaba Cloud OSS, and Tencent Cloud COS) have become core storage infrastructures for enterprises handling massive amounts of data. However, the official native version of DataX does not yet provide a dedicated read / write plugin for the S3 protocol. Currently, the industry primarily uses the following compromise solutions to achieve data synchronization between S3 and other data sources:

[0004] Firstly, an alternative is to use an HDFS plugin: Utilizing DataX's existing HDFS plugins (hdfsreader / hdfswriter), S3 is accessed via the s3a: / / protocol. This solution is highly dependent on the Hadoop ecosystem, requiring the introduction of numerous redundant Hadoop dependency packages. Furthermore, reading and writing are performed through the S3A FileSystem protocol conversion layer, leading to reduced data transfer efficiency, increased memory consumption, and a higher risk of connection timeouts or OutOfMemory (OOM) errors.

[0005] Secondly, a local disk is used as an intermediary: the S3 file is first downloaded to the local disk via the AWS CLI or a custom script, and then read by DataX's txtfilereader; the reverse is done for writing. This solution introduces additional disk I / O overhead, requires a large amount of local storage space, cannot achieve end-to-end streaming synchronization, and is difficult to meet the requirements of high concurrency and high real-time performance.

[0006] In summary, there is an urgent need to develop a new DataX S3 data synchronization solution that does not rely on a Hadoop environment, performs streaming read and write directly based on the S3 API, supports high concurrency, and has strong fault tolerance. Summary of the Invention

[0007] To overcome the aforementioned deficiencies in the existing technology, this application proposes a novel S3 data synchronization method and apparatus based on DataX. More specifically, this invention proposes a method and apparatus for efficient data synchronization between Amazon S3 (SimpleStorage Service) or S3-compatible object storage and heterogeneous data sources based on the DataX framework.

[0008] This invention enables streaming data reading and writing by building native s3reader and s3writer plugins and directly calling the S3 SDK, thus eliminating the dependence on the Hadoop environment. It also introduces dynamic task splitting, double-buffered streaming writing, and sharded upload optimization mechanisms to achieve efficient, stable, and low-latency data synchronization between S3 and heterogeneous data sources.

[0009] To achieve the above objectives, the present invention employs the following technical strategies:

[0010] 1. Seamlessly Embedded DataX "Bidirectional S3 Adapter" Architecture Design: Addressing the issue that DataX does not natively support S3, and the industry commonly uses "HDFS transition method" or "local relay method," which results in long links and low efficiency, this invention is the first to design S3Reader and S3Writer core plugins that fully comply with the DataX plugin specification. By rewriting the Job and Task interfaces, S3-specific APIs (such as S3 Select and Multipart Upload) are seamlessly connected with DataX's internal data exchange channels (Record / Channel), enabling direct interoperability between heterogeneous data sources and S3 without any intermediate medium.

[0011] 2. Zero-Disk-Dependency Streaming Mechanism Based on Double Buffers: Addressing the issues of traditional synchronous methods that require downloading large S3 files to the local disk before DataX reads them, or caching write data locally before uploading, which easily leads to disk overload and significant I / O overhead, this invention proposes a zero-disk-dependency solution for the entire data transfer chain.

[0012] Reader: Utilizes S3's GetObject streaming input stream, combined with multithreading, to directly parse the network stream into a DataXRecord.

[0013] On the write end: A "double buffer" mechanism was designed. One buffer receives data from the DataX channel, while the other buffer asynchronously performs multipart uploads to S3. The two buffers work alternately, realizing a memory-level pipeline operation, which greatly improves throughput.

[0014] 3. Task "Dynamic Adaptive Splitting" Algorithm Based on S3 Metadata: Addressing the issue that S3, as an object storage system, does not support physical partitioning like traditional databases, making it difficult to perform reasonable concurrent task splitting and prone to data skew or concurrency conflicts, this invention proposes a dynamic splitting algorithm tailored to the characteristics of S3:

[0015] During the Job phase, the S3 API is used to quickly obtain the object metadata (file size, number, prefix) under the target path.

[0016] Based on the number of channels configured by the user (Byte / Record rate limit) and the S3 single-shard limit (5MB-5GB), the optimal splitting granularity is dynamically calculated, large files are split into multiple Byte Range tasks, small files are merged, and a balanced TaskGroup is generated, maximizing the use of DataX's concurrency framework.

[0017] 4. Distributed Transaction Consistency Control Mechanism Based on S3 Shard Upload: To address the issue of dirty data (residual incomplete shards) or data inconsistency arising from the failure of a concurrent thread during large-scale data writing to S3, this invention deeply integrates S3's Multipart Upload protocol with DataX's lifecycle management.

[0018] During task execution, each concurrent Task only uploads the shard and obtains the PartETag; no visible object is generated on the S3 side.

[0019] Only when all tasks have completed successfully is a one-time Complete MultipartUpload commit initiated by the Post-Job phase, achieving a "quasi-atomic" commit.

[0020] If any Task fails, the Abort Multipart Upload automatic cleanup mechanism is triggered, which completely avoids residual garbage data and ensures strong consistency.

[0021] Specifically, this application provides the following technical solutions:

[0022] The first aspect of this application provides an S3 data synchronization method based on DataX, such as... Figure 1 As shown, the method includes the following steps:

[0023] S1. Initialization configuration and connection construction: Parse the configuration file of the DataX job, obtain the connection parameters of the S3 data source, and build a thread-safe S3 client instance during the initialization phase of the DataX Reader / Writer plugin;

[0024] S2. Dynamic task splitting: On the reading end, the S3 client traverses the target objects according to the configured path prefix, and uses a dynamic splitting algorithm based on file size balancing to divide the list of objects to be read into multiple task fragments based on object metadata and concurrency. On the writing end, a corresponding number of temporary writing channels are generated under the target path according to the number of concurrent channels, and each channel corresponds to an independent fragment upload task.

[0025] S3, streaming data reading and parsing: In the Reader Task, a GetObject request is initiated through the S3 client to obtain the file input stream. The streaming parser reads the network stream and parses it into Record records inside DataX, and then sends it to the Writer end through the channel.

[0026] S4, double-buffered streaming write and chunked upload: In the Writer Task, Record records are obtained from the channel and serialized into a data stream of the target format. A double-buffered mechanism is used to accumulate data in memory. When the buffer reaches the set threshold, the UploadPart operation of S3 is asynchronously triggered to upload the data block to S3.

[0027] S5. Transactional Commit and Exception Cleanup: After all Writer Tasks have finished writing, the CompleteMultipartUpload interface of S3 is called to merge all fragments and generate the final S3 object. If an exception occurs during the synchronization process, the AbortMultipartUpload interface is called to clean up the uploaded temporary fragments.

[0028] Furthermore, in the method of this application, the connection parameters of the S3 data source in step S1 include: AccessKey, SecretKey, Endpoint, Region, Bucket name, file path prefix, file format and encoding.

[0029] Furthermore, in the method of this application, the dynamic segmentation algorithm of the reading end in step S2 includes: parsing wildcards in the path parameters, listing the matching S3 objects through ListObjectsV2Request, obtaining the Key and Size of each object, calculating the total file size and the average target size of each Task, traversing the object list to allocate the objects to different Task configurations, so that the cumulative file size allocated to each Task is balanced.

[0030] Furthermore, in the method of this application, each Task on the writing end in step S2 sends an InitializeMultipartUpload request to S3 at startup, obtains uploadId and records it in the Task context.

[0031] Furthermore, in the method of this application, the streaming parser described in step S3 uses a BufferedReader to wrap the input stream for CSV / Text format and reads it line by line, splitting the fields according to the specified delimiter and converting them into DataX Column type to construct a Record; for Parquet / ORC format, it uses a memory file system to wrap the S3 stream as a SeekableInputStream and reads and parses it by RowGroup.

[0032] Furthermore, in the method of this application, the dual-buffer mechanism described in step S4 includes: opening two byte array buffers of equal size in memory, and an asynchronous upload thread pool; data is written to the currently active buffer, and when the buffer is full, it is marked as pending upload and switched to the other buffer to continue receiving data; the background thread pool packages the data in the pending upload buffer into a ByteArrayInputStream, calls uploadPart to upload, and records PartETag; the two buffers work alternately to realize memory-level pipeline operation.

[0033] Furthermore, in the method of this application, the buffer size is 8MB, and the size of each fragment except the last fragment meets the minimum size requirement for S3 fragment upload.

[0034] Furthermore, in the method of this application, the transactional commit in step S5 includes: uploading the currently incomplete buffer as the last fragment, waiting for all asynchronous upload tasks to complete, collecting all PartETag and sorting them by fragment number, and calling CompleteMultipartUpload to complete the merging;

[0035] The exception cleanup includes: when any fragment upload fails and the retry limit is exceeded or the DataX framework reports an error, the Task immediately calls AbortMultipartUpload to delete all uploaded temporary fragments after catching the exception.

[0036] Furthermore, in the method of this application, before step S1, it also includes: checking whether the target Bucket exists in the prepare stage of s3writer.Job, and if it is configured to overwrite, deleting the object with the same name or cleaning up the historical fragments.

[0037] A second aspect of this application provides an S3 data synchronization device based on DataX, wherein the device, when operating, implements the steps of the aforementioned S3 data synchronization method based on DataX, such as... Figure 2 As shown, the device includes:

[0038] The configuration parsing module is used to parse the JSON configuration file of the synchronization task and extract S3 connection information, read and write rules and concurrency parameters;

[0039] The connection management module is used to build and manage S3 client connection pools based on the AWS SDK;

[0040] The task splitting module is used to dynamically calculate and split data synchronization tasks based on S3 object metadata and concurrency before the job starts;

[0041] The streaming module is used to directly read data from the S3 input stream and convert it into the DataX standard Record format.

[0042] The streaming write module is used to receive Record data and write the data to S3 using double buffer and fragmented upload technology;

[0043] The transaction control module is used to control the merging and committing of fragmented uploads or the rollback in case of an anomaly.

[0044] Furthermore, in the device of this application, the streaming write module includes a dual buffer unit and an asynchronous upload thread pool. The dual buffer unit contains two memory buffers, one buffer receives data from the DataX channel, and the other buffer asynchronously uploads data in chunks to S3. The two buffers work alternately.

[0045] The asynchronous upload thread pool is used to asynchronously upload data from a full buffer to S3.

[0046] A third aspect of this application provides an electronic device, including: a memory and a processor;

[0047] Memory: Used to store computer programs;

[0048] Processor: Used to execute the computer program to implement the steps of the aforementioned DataX-based S3 data synchronization method.

[0049] A fourth aspect of this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the aforementioned DataX-based S3 data synchronization method.

[0050] In summary, compared with the prior art, the solution of the present invention has the following advantages:

[0051] (1) No Hadoop dependency, lightweight and efficient: It is developed directly based on the native S3 SDK, without loading Hadoop-related JAR packages, which reduces system complexity and memory usage by more than 50%.

[0052] (2) Streaming transmission, zero disk I / O: The reading and writing processes are all streamed, and the data does not need to be written to the local disk, which greatly improves the synchronization rate and reduces the hardware cost.

[0053] (3) Dynamic load balancing: By estimating and dynamically splitting the size of S3 objects, the "long tail effect" caused by uneven file size is avoided, so that the multi-channel concurrent performance can be optimized.

[0054] (4) High reliability and transaction support: The S3 multipart upload mechanism is used to realize breakpoint resume and transactional commit, ensuring the "all or nothing" of data synchronization and avoiding the generation of dirty data.

[0055] Other features and advantages of this application will be set forth in detail in the following description, or will become apparent through the implementation of the relevant technical solutions of this application. The objectives and other advantages of this application can be achieved through the technical features and means explicitly pointed out in the description, claims, and drawings, and will be obtained through the implementation of these technical contents. Attached Figure Description

[0056] To more clearly illustrate the technical solution of this application, the accompanying drawings involved in the description of this invention will be briefly introduced below. It should be noted that the drawings only show some embodiments of the invention. For those skilled in the art, other related drawings can be derived from these drawings without creative effort.

[0057] Figure 1 This is a schematic diagram illustrating the operation steps of the S3 data synchronization method based on DataX in this application.

[0058] Figure 2 This is a structural diagram of the S3 data synchronization device based on DataX in this application.

[0059] Figure 3 This is a flowchart illustrating the overall implementation process of the method in the embodiments of this application.

[0060] Figure 4 This is a logical schematic diagram of the s3reader task dynamic segmentation algorithm in the method of the embodiments of this application.

[0061] Figure 5 This is a flowchart of the s3writer double-buffered streaming write method in the embodiment of this application.

[0062] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application.

[0063] Caption: Processor-310, Communication Interface-320, Memory-330, Communication Bus-340. Detailed Implementation

[0064] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. It should be understood that the described embodiments are only some embodiments of this application, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the protection scope of this application.

[0065] In this document, the term "comprising" and any variations thereof (such as "including," "including," etc.) are open-ended expressions and should be understood as "including but not limited to," meaning that the listed content is not exhaustive and may include other content not explicitly mentioned. The term "based on" should be understood as "at least partially based on," meaning that the basis or condition referred to may not be the only factor and may involve other relevant factors. The term "one embodiment" should be understood as "at least one embodiment," meaning that the described embodiment is not the only possible implementation, and other similar embodiments may exist.

[0066] In this application, the terms "a" and "a plurality of" are used to modify related elements or features, and their expression is illustrative rather than restrictive. Unless otherwise expressly stated in the context, "a" should be understood as "at least one," and "a plurality of" should be understood as "at least two." Those skilled in the art should reasonably interpret these terms based on the semantic and logical relationships of the context to ensure that they cover the possibility of "one or more."

[0067] This invention provides the following technical solution:

[0068] Firstly, the present invention provides an S3 data synchronization method based on DataX. Figure 3 The overall implementation flowchart of the method of the present invention includes the following steps:

[0069] Step S1: Initialization Configuration and Connection Establishment

[0070] After the DataX job starts, it parses the user-configured JSON job file to obtain the connection parameters of the S3 data source (including AccessKey, SecretKey, Endpoint, Region, Bucket name, file path prefix, file format and encoding, etc.), and builds a thread-safe S3 client instance during the DataX Reader / Writer plugin initialization phase.

[0071] Step S2: Dynamic Task Split

[0072] For the reader (s3reader): The S3 client traverses all target objects under the Bucket according to the configured path prefix. Based on the configured number of channels and file size, a dynamic splitting algorithm based on file size balancing is used to divide the list of objects to be read into multiple task groups. Each task group contains one or more S3 objects, ensuring load balancing across channels.

[0073] For the writer (s3writer): Based on the number of channels configured by the user, a corresponding number of temporary write channels are generated under the S3 target path, and each channel corresponds to an independent S3 multipart upload task.

[0074] Step S3: Streaming data reading and parsing (Reader Task execution)

[0075] In each Reader Task, a GetObject request is initiated through the S3 client to obtain the file input stream (InputStream). A zero-copy streaming parser is used to parse the network stream into DataX internal Records while reading the specified file format (such as CSV, JSON, Parquet, ORC, etc.), and then send it to the Writer end through a channel.

[0076] Step S4: Double-buffered streaming write and chunked upload (Writer Task execution)

[0077] In each Writer Task, Records are retrieved from the channel in a loop and serialized into a data stream of the target format. A double buffer mechanism is used to accumulate data in memory. When the buffer reaches a set threshold (e.g., 8MB), the UploadPart operation of S3 is triggered asynchronously to upload the data block to S3.

[0078] Step S5: Transactional Commit and Cleanup

[0079] Once all Writer Tasks have finished writing, the main process calls the S3 CompleteMultipartUpload interface to merge all fragments and generate the final S3 object. If an exception occurs during synchronization, the AbortMultipartUpload interface is called to clean up the uploaded temporary fragments, ensuring the atomicity of data writing.

[0080] Secondly, the present invention provides an S3 data synchronization device based on DataX, comprising:

[0081] 1. Configuration parsing module: Used to parse the JSON configuration file of the synchronization task and extract S3 connection information, read and write rules and concurrency parameters.

[0082] 2. Connection Management Module: Used to build and manage S3 client connection pools based on the AWS SDK.

[0083] 3. Task Splitting Module: Used to dynamically calculate and split data synchronization tasks based on S3 object metadata and concurrency before the job starts.

[0084] 4. Streaming Reading Module: Used to directly read data through the S3 input stream and convert it into the DataX standard Record format.

[0085] 5. Streaming write module: Used to receive Record data and write the data to S3 using double buffer and fragmented upload technology.

[0086] 6. Transaction control module: Used to control the merging and committing of fragmented uploads or the rollback in case of an error.

[0087] To more clearly illustrate the technical solution of this application, the following will provide further explanation through specific scenario embodiments.

[0088] The detailed implementation process of the present invention is as follows:

[0089] 1. Plug-in architecture design

[0090] This invention builds two core plugins within the DataX framework: s3reader and s3writer. Their core class structure is as follows:

[0091] s3reader: Inherits from the Reader base class of DataX. It contains two inner static classes: Job and Task.

[0092] s3writer: Inherits from the Writer base class of DataX. It also contains two inner static classes: Job and Task.

[0093] 2. Detailed implementation process of s3reader

[0094] The synchronization process of s3reader is as follows:

[0095] Step 2.1: Job Initialization and Configuration Analysis

[0096] In the `init()` method of `s3reader.Job`, read and validate the following parameters:

[0097] {

[0098] "endpoint": "s3.amazonaws.com",

[0099] "accessKey": "xxx",

[0100] "secretKey": "xxx",

[0101] "bucket": "my-data-bucket",

[0102] "path": [" / user / behavior / 20231027 / *"],

[0103] "column": [{"index": 0, "type": "string"}, {"index": 1, "type": "long"}],

[0104] "encoding": "UTF-8"

[0105] }

[0106] Use AmazonS3ClientBuilder to build an S3 client instance.

[0107] Step 2.2: Dynamic Task Split

[0108] Figure 4 This is a logical diagram of the s3reader task dynamic segmentation algorithm in this invention.

[0109] In the s3reader.Job.split(int adviceNumber) method:

[0110] (1) Wildcard parsing: Parses the path parameter, supporting wildcards (such as *, ?). Lists all matching S3 objects using S3's ListObjectsV2Request.

[0111] (2) Get metadata: Get the Key and Size (file size) of each object.

[0112] (3) Load balancing and splitting:

[0113] Calculate the total file size (TotalSize).

[0114] Calculate the average size of each Task: TargetSize = TotalSize / adviceNumber.

[0115] Iterate through the list of objects and assign them to different Task configurations, so that the cumulative size of the files assigned to each Task is as close as possible to TargetSize.

[0116] Return List <configuration>Each Configuration contains a list of S3 object keys that the Task is responsible for reading.

[0117] Step 2.3: Task Execution and Streaming Parsing

[0118] In s3reader.Task.startRead(RecordSender recordSender):

[0119] (1) Iterate through the list of S3 objects assigned to the current Task.

[0120] (2) For each object, call s3Client.getObject(new GetObjectRequest(bucket,key)) to get S3ObjectInputStream.

[0121] (3) Stream parsing: Instead of loading the file into memory all at once, the input stream is wrapped in a buffered stream (BufferedReader).

[0122] If the format is CSV / Text: Read by line, split the fields using the specified delimiter (e.g., ,), convert to DataX StringColumn, LongColumn, etc., build a Record, and call recordSender.sendToWriter(record).

[0123] If it is in Parquet / ORC format: use a memory file system (such as a lightweight implementation of Hadoop-VFS) to wrap the S3 stream as a SeekableInputStream, and read and parse it by RowGroup.

[0124] 3. Detailed implementation process of s3writer

[0125] The synchronization process of s3writer is as follows:

[0126] Step 3.1: Job Initialization and Preparation

[0127] In `s3writer.Job.init()`, the S3 client is initialized. In the `prepare()` method, it checks if the target bucket exists. If truncate is configured, objects with the same name are deleted or historical fragments are cleaned up at this stage.

[0128] Step 3.2: Task Segmentation and Initialization of Fragment Upload

[0129] In s3writer.Job.split(int mandatoryNumber), the corresponding number of Tasks are directly split based on mandatoryNumber (number of concurrent channels).

[0130] When each Task starts, it sends an initial fragment upload request to S3:

[0131] InitiateMultipartUploadRequest initRequest = newInitiateMultipartUploadRequest(bucket, targetKey);

[0132] InitiateMultipartUploadResult initResponse = s3Client.initiateMultipartUpload(initRequest);

[0133] String uploadId = initResponse.getUploadId();

[0134] Record the uploadId in the Task context.

[0135] Step 3.3: Double-buffered streaming write (core design point)

[0136] Figure 5 This is a flowchart of the s3writer double-buffered streaming write process in this invention.

[0137] In `s3writer.Task.startWrite(RecordReceiver recordReceiver)`, because DataX records are pushed one by one, and S3's UploadPart requires each fragment (except the last one) to be at least 5MB in size, a double-buffer mechanism is designed:

[0138] Define a double buffer: Allocate two 8MB byte array buffers (Buffer_A and Buffer_B) in memory, and an asynchronous upload thread pool.

[0139] Data writing and switching:

[0140] Get the Record from the RecordReceiver, serialize it into a byte stream (such as row data in CSV format), and write it to the currently active Buffer_A.

[0141] When Buffer_A is full (reaching 8MB), Buffer_A is marked as "pending upload" and the active buffer is switched to Buffer_B to continue receiving Records.

[0142] The background thread pool starts immediately, wraps the data in Buffer_A into a ByteArrayInputStream, calls s3Client.uploadPart to upload it to S3, and records the returned PartETag.

[0143] When Buffer_B is full, switch back to Buffer_A (at this point, Buffer_A has been uploaded and cleared), and repeat the process.

[0144] Zero-copy optimization: The entire process is transmitted directly between the memory buffer and the S3 network stream, without any local disk writes.

[0145] Step 3.4: Transactional Commit

[0146] When recordReceiver has no data:

[0147] Upload the currently unfilled buffer (e.g., 3MB remaining) as the last part.

[0148] Wait for all asynchronous upload tasks in the thread pool to complete, collect all PartETags and sort them by PartNumber.

[0149] Calling s3Client.completeMultipartUpload(newCompleteMultipartUploadRequest(bucket, targetKey, uploadId, partETags)) merges all fragments into a single complete file, making the data publicly available.

[0150] Exception handling (Abort): If any part upload fails and the retry limit is exceeded during the write process, or if the DataX framework reports an error, the Task will immediately call s3Client.abortMultipartUpload(newAbortMultipartUploadRequest(bucket, targetKey, uploadId)) after catching the exception. S3 will automatically delete all uploaded temporary parts to ensure that no garbage files remain.

[0151] The flowcharts and block diagrams in the accompanying drawings illustrate possible implementations of apparatus, methods, and computer program products according to various embodiments of this application, including architecture, functionality, and operation. In these figures, each block may represent a module, program segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should be noted that each block in the block diagrams and / or flowcharts, and combinations thereof, can be implemented using either dedicated hardware-based means or a combination of dedicated hardware and computer instructions to achieve the specified function or operation.

[0152] like Figure 6 As shown, embodiments of this application also disclose an electronic device, including: a processor 310, a communication interface 320, a memory 330 for storing a processor-executable computer program, and a communication bus 340. The processor 310, communication interface 320, and memory 330 communicate with each other via the communication bus 340. The processor 310 executes the executable computer program to implement the steps of the DataX-based S3 data synchronization method described above.

[0153] It is understood that, in addition to memory and a processor, this electronic device may also include input devices (such as a keyboard), output devices (such as a display), and other communication modules. These input devices, output devices, and other communication modules all communicate with the processor through I / O interfaces (i.e., input / output interfaces).

[0154] The operations described in this application can be implemented by writing computer program code using one or more programming languages ​​or a combination thereof. The programming languages ​​include, but are not limited to, the following types:

[0155] Object-oriented programming languages, such as Java, Smalltalk, C++, etc.

[0156] Conventional procedural programming languages, such as "C" or similar programming languages.

[0157] The execution methods of program code include, but are not limited to:

[0158] It runs entirely on the user's computer;

[0159] Part of it executes on the user's computer, and part of it executes on a remote computer;

[0160] Execute as a standalone software package;

[0161] It is executed entirely on a remote computer or server.

[0162] In scenarios involving remote computers, the remote computer can connect to the user's computer via any type of network, including but not limited to local area networks (LANs) or wide area networks (WANs). Furthermore, the remote computer can also connect to external computers through an internet service provider, for example, by utilizing the internet for connection.

[0163] Furthermore, this application also discloses a computer-readable storage medium, which, when the instructions in the computer-readable storage medium are executed by a processor of an electronic device, enables the electronic device to perform the various steps of the DataX-based S3 data synchronization method disclosed in this application.

[0164] In the context of this application, a computer-readable storage medium refers to a tangible medium capable of storing computer program code and related data. Specific examples include, but are not limited to, the following:

[0165] (1) Portable computer disk: such as floppy disks and other removable magnetic storage media.

[0166] (2) Hard disk: including mechanical hard disks and solid-state hard disks and other fixed storage devices.

[0167] (3) Random Access Memory (RAM): A volatile storage medium used for temporary storage of data and program code.

[0168] (4) Read-only memory (ROM): a non-volatile storage medium used to store fixed programs and data.

[0169] (5) Erasable programmable read-only memory (EPROM) or flash memory: non-volatile storage media that supports multiple erasures and reprogrammings.

[0170] (6) Fiber optic storage devices: storage media based on fiber optic technology.

[0171] (7) Portable compact disc read-only memory (CD-ROM): a read-only medium that stores data in the form of an optical disc.

[0172] (8) Optical storage devices: such as DVDs, Blu-ray discs and other storage media based on optical principles.

[0173] (9) Magnetic storage devices: such as magnetic tapes, disks and other storage media based on magnetic principles.

[0174] (10) Any suitable combination of the above: for example, combining multiple storage media to meet different storage needs.

[0175] These computer-readable storage media can be used to store the program code and related data described in this application to support program execution and persistent data storage.

[0176] Specifically, according to embodiments of this application, the processes described in the flowcharts can be implemented as computer software programs. For example, embodiments of this application relate to a computer program product comprising a computer program carried on a non-transitory computer-readable medium. This computer program includes program code for executing the DataX-based S3 data synchronization method disclosed in this application. When the computer program is executed by a processing device, it can achieve the functions defined in the embodiments of this application.

[0177] While the foregoing discussion contains several specific implementation details, these details should not be construed as limiting the scope of this application. The above description is merely a preferred embodiment of this application and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of this application is not limited to technical solutions formed by specific combinations of the above-described technical features. Furthermore, this application should also cover other technical solutions formed by any combination of the above-described technical features or their equivalents without departing from the foregoing disclosed concept.

[0178] Those skilled in the art should also understand that modifications can be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features, without departing from the spirit and scope of the technical solutions of the embodiments of this application. These modifications or substitutions will not cause the essence of the corresponding technical solutions to deviate from the core spirit and scope of the technical solutions of the embodiments of this application.< / configuration>

Claims

1. An S3 data synchronization method based on DataX, characterized in that, Includes the following steps: S1. Initialization configuration and connection construction: Parse the configuration file of the DataX job, obtain the connection parameters of the S3 data source, and build a thread-safe S3 client instance during the initialization phase of the DataX Reader / Writer plugin; S2. Dynamic task splitting: On the reading end, the S3 client traverses the target object according to the configured path prefix, and uses a dynamic splitting algorithm based on file size balancing to divide the list of objects to be read into multiple task slices based on object metadata and concurrency. On the writing side, a corresponding number of temporary write channels are generated under the target path according to the number of concurrent channels, and each channel corresponds to an independent chunked upload task; S3, streaming data reading and parsing: In the Reader Task, a GetObject request is initiated through the S3 client to obtain the file input stream. The streaming parser reads the network stream and parses it into Record records inside DataX, and then sends it to the Writer end through the channel. S4, double-buffered streaming write and chunked upload: In the Writer Task, Record records are obtained from the channel and serialized into a data stream of the target format. A double-buffered mechanism is used to accumulate data in memory. When the buffer reaches the set threshold, the UploadPart operation of S3 is asynchronously triggered to upload the data block to S3. S5. Transactional Commit and Exception Cleanup: After all Writer Tasks have finished writing, the CompleteMultipartUpload interface of S3 is called to merge all fragments and generate the final S3 object. If an exception occurs during the synchronization process, the AbortMultipartUpload interface is called to clean up the uploaded temporary fragments.

2. The method according to claim 1, characterized in that, The connection parameters of the S3 data source mentioned in step S1 include: AccessKey, SecretKey, Endpoint, Region, Bucket name, file path prefix, file format and encoding.

3. The method according to claim 1, characterized in that, The dynamic splitting algorithm at the reading end in step S2 includes: parsing wildcards in the path parameters, listing matching S3 objects through ListObjectsV2Request, obtaining the Key and Size of each object, calculating the total file size and the average target size of each Task, traversing the object list to allocate objects to different Task configurations, so that the cumulative file size allocated to each Task is balanced.

4. The method according to claim 1, characterized in that, In step S2, each Task on the writing end sends an InitializeMultipartUpload request to S3 at startup, obtains the uploadId, and records it in the Task context.

5. The method according to claim 1, characterized in that, The streaming parser described in step S3 uses a BufferedReader to wrap the input stream for CSV / Text format and reads it line by line, splitting the fields according to the specified delimiter and converting them into DataX Column types to construct a Record; for Parquet / ORC format, it uses the memory file system to wrap the S3 stream as a SeekableInputStream and reads and parses it by RowGroup.

6. The method according to claim 1, characterized in that, The dual-buffer mechanism described in step S4 includes: allocating two byte array buffers of equal size in memory, and an asynchronous upload thread pool; writing data into the currently active buffer, marking it as pending upload when the buffer is full and switching to the other buffer to continue receiving data; the background thread pool packages the data in the pending upload buffer into a ByteArrayInputStream, calls uploadPart to upload, and records PartETag; the two buffers work alternately to achieve memory-level pipelined operation.

7. The method according to claim 1, characterized in that, The transactional commit in step S5 includes: uploading the currently incomplete buffer as the last fragment, waiting for all asynchronous upload tasks to complete, collecting all PartETags and sorting them by fragment number, and calling CompleteMultipartUpload to complete the merge; The exception cleanup includes: when any fragment upload fails and the retry limit is exceeded or the DataX framework reports an error, the Task immediately calls AbortMultipartUpload to delete all uploaded temporary fragments after catching the exception.

8. The method according to claim 1, characterized in that, Before step S1, the following steps are also included: in the prepare stage of s3writer.Job, check if the target Bucket exists, and if it is configured to overwrite, delete the object with the same name or clean up the historical fragments.

9. An S3 data synchronization device based on DataX, characterized in that, When the device is in operation, it implements the steps of the S3 data synchronization method based on DataX as described in any one of claims 1-8, including: The configuration parsing module is used to parse the JSON configuration file of the synchronization task and extract S3 connection information, read and write rules and concurrency parameters; The connection management module is used to build and manage S3 client connection pools based on the AWS SDK; The task splitting module is used to dynamically calculate and split data synchronization tasks based on S3 object metadata and concurrency before the job starts; The streaming module is used to directly read data from the S3 input stream and convert it into the DataX standard Record format. The streaming write module is used to receive Record data and write the data to S3 using double buffer and fragmented upload technology; The transaction control module is used to control the merging and committing of fragmented uploads or the rollback in case of an anomaly.

10. The apparatus according to claim 9, characterized in that, The streaming write module includes a dual buffer unit and an asynchronous upload thread pool. The dual buffer unit contains two memory buffers. One buffer receives data from the DataX channel, and the other buffer asynchronously uploads fragmented data to S3. The two buffers work alternately. The asynchronous upload thread pool is used to asynchronously upload data from a full buffer to S3.

Images