A flink-based unstructured file transmission method and system

Abstract

The application discloses a Flink-based unstructured file transmission method and system. The method comprises the following steps: reading a file from object storage and encapsulating the file into a data object through a self-defined Source operator; splitting the data object into a data block stream containing block serial numbers through a flatMap operator; grouping the data block stream based on a file name keyBy, so that data blocks of the same file are routed to the same task instance; initializing an expected block serial number, an out-of-order block cache, a completion flag and a temporary file path through a KeyedProcessFunction, sequentially writing a local temporary file and processing out-of-order blocks, uploading the temporary file to a target object storage after completion, and cleaning up the state. The application fully utilizes the distributed parallel capability and state management mechanism of Flink, realizes efficient parallel transmission, out-of-order recombination, integrity check and breakpoint resume of unstructured files, simplifies the integration with a Flink stream batch integrated architecture, and improves transmission efficiency and reliability.

Description

Technical Field

[0001] This application relates to the field of data synchronization technology, and in particular to a method, system, computer-readable storage medium, and electronic device for unstructured file transfer based on Flink. Background Technology

[0002] With the evolution of cloud computing and big data technologies, the efficient transmission of massive amounts of unstructured data (such as logs, binary files, and media files) has become a key challenge in the field of distributed computing. Currently, the industry generally adopts a stream-batch integrated architecture to simplify data processing workflows. Apache Flink, as a mainstream open-source stream processing engine, is widely used in structured and semi-structured data processing scenarios due to its distributed computing capabilities and low latency characteristics.

[0003] However, the existing Flink ecosystem has significant limitations in supporting unstructured file transfer: First, traditional file transfer solutions (such as those based on FTP / SFTP protocols or native HDFS / S3 interfaces) are poorly coupled with the Flink runtime, requiring the introduction of an additional adaptation layer, which increases the complexity of the system architecture; Second, in Flink's distributed execution environment, data fragments are often randomly distributed across computing nodes and are prone to out-of-order arrival due to network latency, making it difficult for existing mechanisms to guarantee the orderly reassembly of file fragments and end-to-end integrity verification; Third, it lacks fine-grained fault tolerance and breakpoint resumption capabilities for abnormal scenarios such as node failures or network interruptions, resulting in insufficient transmission reliability.

[0004] In summary, existing Flink solutions for transmitting unstructured files are still imperfect, especially when dealing with complex situations such as random and out-of-order data in distributed environments. These solutions suffer from low transmission efficiency, difficulty in guaranteeing data integrity, and insufficient fault tolerance. Therefore, there is an urgent need for an innovative method that can deeply leverage Flink's distributed computing capabilities to achieve efficient and reliable transmission of unstructured files, addressing key technical challenges such as out-of-order reordering, integrity assurance, and fault tolerance recovery. Summary of the Invention

[0005] To overcome the aforementioned shortcomings of existing technologies, this application proposes a novel unstructured file transfer method and system based on Flink. This invention aims to address the problems of low efficiency, difficulty in ensuring data integrity, and insufficient fault tolerance in Flink's handling of unstructured file transfers, thereby achieving ordered and reliable data transfer and fully utilizing Flink's parallel processing capabilities.

[0006] To achieve the above objectives, the present invention employs the following technical strategies:

[0007] (1) Distributed unstructured file splitting and ordered transmission: For the first time, it is proposed to split unstructured files within the Flink distributed stream processing framework. The keyBy operator ensures that data blocks of the same file are aggregated into the same parallel instance for processing, thereby achieving ordered generation of single files.

[0008] (2) State management based on state machine: A state machine model is built in the OrderedFileWriter operator, and state management is implemented with state variables (nextExpectedChunkIndex, chunkBuffer, isFileComplete, tempFilePathState). This can reliably handle out-of-order arrival of data blocks and achieve complete file merging.

[0009] (3) Streaming breakpoint resume mechanism: Combining local temporary file storage and Flink state management, the breakpoint resume function is implemented in a distributed streaming processing environment to meet the needs of large unstructured file transmission.

[0010] (4) Seamless integration with object storage: By customizing the Source and Sink components, direct integration with object storage such as MinIO can be achieved, avoiding additional intermediate storage and complex protocol conversion.

[0011] (5) Unstructured file processing with integrated stream and batch processing: Unstructured file transmission is integrated into the Flink stream processing model, providing support for building an integrated stream and batch data processing flow, and supporting flexible combination with other Flink stream and batch processing operators.

[0012] Specifically, this application provides the following technical solutions:

[0013] The first aspect of this application provides a method for unstructured file transfer based on Flink, such as... Figure 1 As shown, the method includes:

[0014] S1. Read unstructured files from object storage using a custom Source operator, encapsulate the file content into a data object containing the filename and a byte array, and output it downstream;

[0015] S2. The data object is divided into several data blocks using the flatMap operator. Each data block contains a file name, block number, byte data, and end block marker, forming a data block stream.

[0016] S3. Group the data block stream by keyBy based on the filename, so that all data blocks of the same file are routed to the same parallel task instance;

[0017] S4. The KeyedProcessFunction operator is used to perform ordered processing on the grouped data blocks and transmit them to the target end. Specifically, this includes:

[0018] S41. Initialization phase: Create a connection with the target object storage and initialize state variables, including expected block sequence number, out-of-order block cache, file completion flag and temporary file path;

[0019] S42. Processing stage: Receive data blocks. When the current block sequence number is equal to the expected block sequence number, write the current data block to the local temporary file and update the expected block sequence number. Otherwise, store the current data block in the out-of-order block cache.

[0020] S43, Cyclic Check Phase: Check if there are consecutive data blocks in the out-of-order block cache. If they exist, write them to the local temporary file and update the expected block sequence number.

[0021] S44. Completion Phase: After receiving the last block marker and completing the sequential writing of all data blocks, upload the local temporary file to the target object storage and clean up the relevant state variables.

[0022] Furthermore, in the method of this application, the step S1 of reading unstructured files from object storage using a custom Source operator specifically includes:

[0023] Initialize the MinIO client, and configure the server endpoint, access key, and key.

[0024] List objects under the source Bucket based on directory prefixes, filter directory objects, and then traverse the files;

[0025] File byte data is read into memory through the input stream, encapsulated into a FileContent object, and output through the context collector.

[0026] Furthermore, in the method of this application, the data block mentioned in step S2 is a FileChunk object, which splits the file by a preset block size, with the block number incrementing from 0, and the last block is marked as IsLastChunk.

[0027] Furthermore, in the method of this application, step S2, which involves dividing the data object into several data blocks using the flatMap operator, specifically includes:

[0028] Extract filenames and file byte data, and then segment them cyclically according to a preset block size;

[0029] Calculate the actual length of the current block and copy the corresponding bytes of data;

[0030] Determine if it is the last block, generate a FileChunk object containing the filename, incrementing block number, byte data, and last block marker, and output it.

[0031] Furthermore, in the method of this application, the state variables mentioned in step S41 specifically include:

[0032] nextExpectedChunkIndex:ValueState <long>Type, stores the sequence number of the next block to be received, initially set to 0;

[0033] chunkBuffer:MapState<Long, byte[]> Type: Out-of-order block cache; key is block sequence number; value is block data.

[0034] isFileComplete:ValueState <boolean>Type, indicates whether file reception is complete, initially false;

[0035] tempFilePathState:ValueState <string>Type, path to store local temporary files.

[0036] Furthermore, in the method of this application, the step S42 of writing the current data block to a local temporary file specifically includes:

[0037] If the temporary file path status is empty, a unique temporary file name is generated and the status is updated;

[0038] Open the local temporary file in append mode and write the current data block bytes of data.

[0039] Furthermore, in the method of this application, step S43 includes:

[0040] Loop through the out-of-order block cache to check if there is a data block with a sequence number equal to the current expected block sequence number;

[0041] If it exists, write the data block to a local temporary file, remove it from the out-of-order block cache, update the expected block sequence number, and repeat the above process until there are no consecutive sequence numbers.

[0042] Furthermore, in the method of this application, the step S44 of uploading the local temporary file to the target object storage specifically includes:

[0043] Get the local temporary file path and open the file input stream;

[0044] The object storage SDK is invoked to perform an upload operation. After a successful upload, the local temporary file is deleted. The upload operation includes a retry mechanism, which performs a limited number of retries when a recoverable exception occurs.

[0045] Furthermore, the method in this application also includes a breakpoint resume mechanism: by recording the local temporary file path through the temporary file path, and combining the expected block sequence number and the out-of-order block cache state, the file merging and uploading are resumed from the breakpoint after the Flink task restarts.

[0046] A second aspect of this application provides a Flink-based unstructured file transfer system, wherein the system implements the steps of the aforementioned Flink-based unstructured file transfer method during operation, and the system includes:

[0047] The data object generation module is used to read unstructured files from object storage using a custom Source operator, encapsulate the file content into a data object containing the filename and a byte array, and output it downstream.

[0048] The data segmentation module is used to segment the data object into several data blocks using the flatMap operator. Each data block contains a file name, block number, byte data, and end block marker, forming a data block stream.

[0049] The group routing module is used to group the data block stream by keyBy based on the file name, so that all data blocks of the same file are routed to the same parallel task instance.

[0050] The ordered processing module is used to perform ordered processing on the grouped data blocks using the KeyedProcessFunction operator and transmit them to the target end. Specifically, it includes:

[0051] An initialization unit is used to create a connection with the target object storage and initialize state variables, including the expected block sequence number, out-of-order block cache, file completion flag, and temporary file path.

[0052] The processing unit is used to receive data blocks. When the current block sequence number is equal to the expected block sequence number, the current data block is written to the local temporary file and the expected block sequence number is updated. Otherwise, the current data block is stored in the out-of-order block cache.

[0053] The cyclic checking unit is used to check whether there are consecutive data blocks in the out-of-order block cache. If they exist, they are written to the local temporary file and the expected block sequence number is updated.

[0054] The completion unit is used to upload the local temporary file to the target object storage and clean up the relevant state variables after receiving the last block marker and completing the sequential writing of all data blocks.

[0055] A third aspect of this application provides an electronic device, including: a memory and a processor;

[0056] Memory: Used to store computer programs;

[0057] Processor: Used to execute the computer program to implement the steps of the aforementioned Flink-based unstructured file transfer method.

[0058] A fourth aspect of this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the aforementioned Flink-based unstructured file transfer method.

[0059] In summary, compared with the prior art, the present invention has the following technical advantages:

[0060] (1) Improve transmission efficiency: Relying on Flink's distributed computing capabilities, large files are divided into multiple data blocks for parallel transmission and processing; the keyBy operator routes all data blocks of the same file to the same task instance, avoiding cross-task coordination overhead and ensuring orderly processing of single files, thereby improving overall transmission efficiency.

[0061] (2) Ensure data integrity: The chunkBuffer caching mechanism is used to temporarily store out-of-order byte blocks and write them in order to avoid data loss caused by network delays; the isFileComplete flag is used to verify the full data block reception status, and the final upload is only performed after all blocks have been written in order to ensure file integrity.

[0062] (3) Implement breakpoint resume: Based on tempFilePathState, the local temporary file path is recorded, and the transmission progress is recorded by state variables such as nextExpectedChunkIndex and chunkBuffer. After the Flink task restarts due to failure, file merging and uploading can be resumed from the breakpoint, which significantly improves the robustness of the system.

[0063] (4) Simplified Flink integration: The logic for transferring unstructured files is encapsulated within Flink operators, requiring no additional components or complex configurations, and can be seamlessly integrated into the Flink stream and batch architecture.

[0064] (5) Enhance the flexibility of the source and target ends: Taking MinIO as an example, its design concept can be extended to other object storage services or file storage systems, and has good versatility.

[0065] Other features and advantages of this application will be set forth in detail in the following description, or will become apparent through the implementation of the relevant technical solutions of this application. The objectives and other advantages of this application can be achieved through the technical features and means explicitly pointed out in the description, claims, and drawings, and will be obtained through the implementation of these technical contents. Attached Figure Description

[0066] To more clearly illustrate the technical solution of this application, the accompanying drawings involved in the description of this invention will be briefly introduced below. It should be noted that the drawings only show some embodiments of the invention. For those skilled in the art, other related drawings can be derived from these drawings without creative effort.

[0067] Figure 1 This is a flowchart illustrating the overall implementation of the Flink-based unstructured file transfer method of this invention.

[0068] Figure 2 This is a framework diagram of the implementation process of the present invention.

[0069] Figure 3 This is a flowchart illustrating the specific implementation process of the present invention, which involves sequentially writing data blocks in Flink to generate unstructured files on the target end.

[0070] Figure 4 This is a structural diagram of the unstructured file transfer system based on Flink according to the present invention.

[0071] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0072] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. It should be noted that the described embodiments are only some embodiments of this application, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the protection scope of this application.

[0073] In this document, the term "comprising" and any variations thereof (such as "including," "including," etc.) are open-ended expressions and should be understood as "including but not limited to," meaning that the listed content is not exhaustive and may include other content not explicitly mentioned. The term "based on" should be understood as "at least partially based on," meaning that the basis or condition referred to may not be the only factor and may involve other relevant factors. The term "one embodiment" should be understood as "at least one embodiment," meaning that the described embodiment is not the only possible implementation, and other similar embodiments may exist.

[0074] In this application, the terms "a" and "a plurality of" are used to modify related elements or features, and their expression is illustrative rather than restrictive. Unless otherwise expressly stated in the context, "a" should be understood as "at least one," and "a plurality of" should be understood as "at least two." Those skilled in the art should reasonably interpret these terms based on the semantic and logical relationships of the context to ensure that they cover the possibility of "one or more."

[0075] Example: A Flink-based method for unstructured file transfer

[0076] This plan includes the following steps:

[0077] 1. Define the data source (Source)

[0078] Using a custom Source operator, all unstructured files in a target directory are read in batches from a specified object store (such as MinIO). The Source operator lists all objects in a specific directory within a specified Bucket, downloads the complete content of each file into memory one by one, encapsulates it into a FileContent object (a byte array containing the filename and file content), and then sends it to the downstream operator via the ctx.collect() method.

[0079] 2. File Segmentation and Block Encapsulation (Transformation)

[0080] The FileContent object output by the Source operator is processed by a flatMap operator (FileToChunkMapper). This operator divides each complete unstructured file into several byte chunks (FileChunks) according to a preset ChunkSize. Each FileChunk object contains the filename, the chunk's index in the original file, the actual chunk data, and a flag (IsLastChunk) indicating whether it is the last chunk in the file.

[0081] 3. Grouping based on filename (KeyBy)

[0082] The split FileChunk data stream is grouped using the filename (fileName) as the key using the key. This operation ensures that all byte blocks (FileChunks) of the same original file can be routed to the same parallel task instance for centralized processing in subsequent processes, thus guaranteeing the ordered processing of individual files.

[0083] 4. Ordered writing and target-side transmission (KeyedProcessFunction)

[0084] Register a KeyedProcessFunction (OrderedFileWriter) operator on the grouped KeyedStream. This operator is responsible for processing all byte blocks from the same filename and implementing ordered writing and final transmission.

[0085] (1) Initialization State (open() method): During the operator initialization phase, a connection to the target MinIO is created based on the provided MinIO client configuration (Endpoint, Access Key, Secret Key). Simultaneously, the following state variables are initialized for each Key (filename):

[0086] nextExpectedChunkIndex(ValueState <long>): Stores the sequence number of the next block of bytes currently waiting to be received, with an initial value of 0.

[0087] chunkBuffer (MapState<Long, byte[]> ): Used to cache out-of-order byte blocks, where Key is the sequence number of the byte block and Value is the data of the byte block.

[0088] isFileComplete(ValueState) <boolean>): Marks whether the last byte block of the current file has been received to prevent repeated processing before the file transfer is complete.

[0089] tempFilePathState (ValueState <string>): Stores the path to the local temporary files currently being merged, used to resume interrupted downloads when the task restarts.

[0090] (2) Process byte blocks one by one (processElement() method):

[0091] Check file completion status: If isFileComplete is true, ignore the byte block and end processing.

[0092] Process in order: If the sequence number of the currently received byte block chunk.getChunkIndex() is equal to the value of nextExpectedChunkIndex, then the byte block is considered to have arrived in order.

[0093] The `processOrderedChunk()` method is called to write the data of the block of bytes to a local temporary file. If this is the first block, the temporary file is created; otherwise, it is written in append mode.

[0094] After writing the sequentially arriving blocks, the `writeBufferedChunks()` method is immediately called to check if there are any byte blocks in the `chunkBuffer` that are contiguous with the current `nextExpectedChunkIndex`. If so, these buffered byte blocks are written to the local temporary file in sequence, and `nextExpectedChunkIndex` is updated.

[0095] If the current byte block is the last byte block (chunk.isLastChunk() is true), then update the isFileComplete status to true.

[0096] Out-of-order buffered blocks: If the sequence number of the currently received byte block is not equal to nextExpectedChunkIndex, the byte block is considered to have arrived out of order, and its data is stored in the chunkBuffer for subsequent sequential writing.

[0097] (3) File completion and upload (logic in processElement()):

[0098] If the currently received chunk.isLastChunk() is true, and after writeBufferedChunks completes, the next expected chunk index nextExpectedChunkIndex is exactly equal to the current chunk index currentChunkIndex + 1, then perform the following operations:

[0099] a) Call the uploadLocalFileToMinio() method to upload the merged temporary file locally to the target MinIO Bucket. This method includes a simple retry mechanism to handle network fluctuations.

[0100] b) After successful upload, delete the local temporary files.

[0101] c) Call the cleanupState() method to clean up the Flink state associated with the current key (filename) and prepare for the processing of the next file.

[0102] Auxiliary methods:

[0103] processOrderedChunk(fileName, chunk, out): Writes the sequentially arriving chunk of bytes to a local temporary file.

[0104] writeBufferedChunks(fileName, out): Loop through the chunkBuffer until consecutive blocks of bytes are written to the local file.

[0105] createTempFile(fileName): Generates a safe and unique local temporary filename.

[0106] uploadLocalFileToMinio(fileName, tempFilePath, out): Uploads a local file to the target Bucket using the MinIO SDK.

[0107] cleanupState(key): Clears the state of the specified key.

[0108] To more clearly illustrate the technical solution of this application, the following will provide further explanation through specific scenario embodiments.

[0109] This invention aims to build an efficient and reliable unstructured file transfer system based on Apache Flink, capable of reading files from object storage (such as MinIO), splitting, ordering, merging, and finally transferring them to the target object storage. The entire process fully utilizes Flink's distributed features and state management capabilities.

[0110] Figure 2 The diagram shown is a framework diagram of the implementation process of the present invention. Figure 3 The figure shows a flowchart illustrating the specific implementation process of this invention, which involves sequentially writing data blocks in Flink to generate unstructured files on the target side. The detailed implementation process of this solution is as follows:

[0111] Step 1: Define the data source (Source) — Read files from MinIO

[0112] 1. Initialize the MinIO client: When a Flink job starts, the `open()` method of `MinioFileSource` is called. In this method, a `MinioClient` instance is constructed based on the user-provided MinIO server address (Endpoint), access key, and secret key. This client object is the entry point for subsequent interactions with MinIO and is created in the `open()` method to ensure its availability throughout the operator's lifecycle, avoiding re-creation for each data processing iteration and improving efficiency.

[0113] 2. Listing files in the source directory: The `run()` method of `MinioFileSource` is where the actual data is generated. It first defines a `directoryPath` (passed as a parameter), which is the directory to be scanned in the MinIO source bucket. Then, it uses the `minioClient.listObjects()` method, along with `ListObjectsArgs`, specifying the bucket name to query, the directory prefix, and `recursive(false)` (meaning no recursion to subdirectories). This returns an iterator containing all objects (files and possibly empty directory objects) in that directory.

[0114] 3. Iterate through and download file contents: Iterate through each Item (object) returned by listObjects().

[0115] Filtering directories: In MinIO, directories themselves may exist as object names ending with " / ". Therefore, you need to skip these directory objects by checking `item.objectName().endsWith(" / ")` and only process the actual files.

[0116] Downloading files: For each file object, use the minioClient.getObject() method in conjunction with GetObjectArgs, specifying the Bucket name and the fully qualified object name (objectPath) of the file, to obtain the file's input stream (InputStream).

[0117] Reading byte data: Create a ByteArrayOutputStream (a byte stream in memory) and set a buffer (e.g., 4096 bytes). Read byte data from the InputStream and write it to the ByteArrayOutputStream in a loop until all bytes have been read (is.read() returns -1).

[0118] Sending a FileContent object: Once the entire contents of a file have been read into a ByteArrayOutputStream, a FileContent object is created, containing the filename (objectPath) and the entire byte content of the file (baos.toByteArray()). Then, this FileContent object is sent to the downstream operator of the Flink data stream using ctx.collect(new FileContent(...)).

[0119] 4. Exception Handling: During file reading, if any exception occurs (such as network problems or permission issues), an error log will be printed, but other files will be processed as much as possible to improve overall availability. If an exception occurs while listing objects, it will be re-thrown, which may trigger Flink's retry mechanism.

[0120] Step 2: Define the transformation operation—splitting the file into byte blocks.

[0121] 1. Receiving FileContent: The flatMap() method of FileToChunkMapper receives the FileContent object output by MinioFileSource in the previous step.

[0122] 2. Obtain file information: Extract the file name (fileName) and the complete byte content (fileData) of the file from the FileContent object.

[0123] 3. Calculate total length and initialize: Get the total length of the file (totalLength) and initialize a chunkIndex to 0 to mark the sequence number of the byte block currently being generated.

[0124] 4. Looping and splitting: Use a for loop, with offset (the starting position of the current byte block) as the variable, and increment FileChunk.CHUNK_SIZE (e.g., 64KB) each time, until offset reaches or exceeds totalLength.

[0125] Calculate the actual length of the current block: length = Math.min(FileChunk.CHUNK_SIZE, totalLength - offset), ensuring that the last byte block does not exceed the end of the file.

[0126] Copying byte data: Create a new byte array `chunkData` with length `length`. Use `System.arraycopy()` to copy the portion of the original file data corresponding to the current block into `chunkData`.

[0127] To determine if it is the last block: isLastChunk = (offset + FileChunk.CHUNK_SIZE) >= totalLength, is used to determine if the currently split byte block is the last block of the file.

[0128] Sending a FileChunk object: Create a FileChunk object containing the filename (fileName), an incrementing chunk index (chunkIndex), the copied byte array (chunkData), and a flag indicating whether it is the last chunk (isLastChunk). Then, send this FileChunk object downstream using `collector.collect(new FileChunk(...))`.

[0129] Step 3: Use keyBy to group by filename

[0130] 1. Grouping logic: The line of code `chunkStream.keyBy(chunk -> chunk.fileName)` is crucial. It tells Flink that all subsequent operations on this data stream must be performed using the `fileName` field of the `FileChunk` object as the key.

[0131] 2. Data Routing: Flink's keyBy operation routes all FileChunk data with the same key (i.e., the same filename) to the same concurrent task instance (subtask) within the same Flink Task Manager for processing. This means that all byte blocks belonging to the same original file will be received and processed by the same OrderedFileWriter operator instance, thus ensuring the sequential processing of a single file.

[0132] Step 4: Ordered writing and transmission to the target end (OrderedFileWriter operator)

[0133] This is the core of the entire solution; it is a KeyedProcessFunction that processes all FileChunks with the same filename that are dispatched to the same task by keyBy.

[0134] 1. Initialization and state setting (open() method)

[0135] MinIO Client: Initialize the target MinIO's MinioClient in the open() method.

[0136] State variable descriptor:

[0137] nextExpectedChunkIndex(ValueState <long>): This value stores the sequence number of the next block of bytes currently waiting to be received. It is initially set to 0. This value increments as blocks arrive in sequence and are processed.

[0138] chunkBuffer (MapState<Long, byte[]> This is a Map type state used to cache out-of-order byte blocks. The key is the sequence number of the byte block, and the value is the byte data of that byte block.

[0139] isFileComplete(ValueState) <boolean>): A boolean value indicating whether the last block of bytes has been received in the current file. The initial value is false.

[0140] tempFilePathState (ValueState <string>): Stores the full path to the local temporary file currently being merged. This state is crucial for enabling resumeable downloads.

[0141] State acquisition: Get instances of these states for each Key (filename) using getRuntimeContext().getState().

[0142] 2. Processing a single byte block (processElement() method)

[0143] When the OrderedFileWriter operator receives a FileChunk, the processElement() method is called.

[0144] Get the filename and current block information: Get the filename of the currently processed block from ctx.getCurrentKey(), and get the block number (currentChunkIndex), data (chunkData), and whether it is the last block (isLastChunk) from the chunk object.

[0145] To check if a file has been completed: First, check the isFileComplete status. If it is already set to true, it means the file has been processed and uploaded, so ignore the currently received block and return directly.

[0146] Get the expected block number: Get the status value of nextExpectedChunkIndex. If there is no value (the first time the block of this file is received), the default value is 0.

[0147] (1) Process in sequence (the current arrival data block number currentChunkIndex equals the expected data block number expectedIndex):

[0148] 1) Write to a local temporary file: Call the processOrderedChunk() method.

[0149] a) Create a temporary file (if needed): If tempFilePathState is empty, it means this is the first block of the file. Call createTempFile() to generate a unique local temporary filename (e.g., using the original filename plus a UUID, stored in the local temporary directory of the Flink Task Manager), and update the path in tempFilePathState.

[0150] b) Append Write: Using Flink's Path and FileSystem APIs, open the corresponding local temporary file in APPEND mode and write the current chunk.getChunkData(). Using BufferedOutputStream can improve write efficiency.

[0151] c) Update the expected chunk index: After a successful write, update nextExpectedChunkIndex to chunk.getChunkIndex() + 1.

[0152] 2) Check and write to the buffer (writeBufferedChunks()): Immediately after processing the sequentially arriving blocks, writeBufferedChunks() is called. This method checks if the chunkBuffer contains a byte block with an index equal to the current nextExpectedChunkIndex. If so, it writes them sequentially to a local temporary file and updates nextExpectedChunkIndex, repeating this process until a sequence number not present in the buffer is encountered, or the buffer is empty.

[0153] 3) File processing complete: If the currently received chunk.isLastChunk() is true, and after writeBufferedChunks is completed, the next expected chunk index nextExpectedChunkIndex is exactly equal to the current chunk index currentChunkIndex + 1, then:

[0154] Update the isFileComplete status to true.

[0155] Get the local temporary file path from tempFilePathState.

[0156] Call the uploadLocalFileToMinio() method to upload the merged local file to the target MinIO.

[0157] Finally, the cleanupState() method is called to clear all states of the current key (filename) and prepare for processing the next file.

[0158] (2) Buffering out-of-order blocks (the current arrival data block number currentChunkIndex is not equal to the expected data block number expectedIndex):

[0159] If currentChunkIndex is not equal to expectedIndex, it means that this is a block that arrived out of order.

[0160] The byte data of this block is stored in the chunkBuffer state using chunk.getChunkData(), with the key being currentChunkIndex.

[0161] Auxiliary method description:

[0162] `processOrderedChunk()`: This function is responsible for actually writing sequentially arriving chunks of bytes to the local temporary file specified by `tempFilePathState`. It handles file creation (if it's the first time writing) and appending.

[0163] `writeBufferedChunks()`: This is a crucial loop logic. It continuously checks if a chunk with the expected index `expectedIndex` (the expected data chunk index) exists in the `chunkBuffer`. If it exists, it retrieves the chunk, calls `processOrderedChunk` to write it to the file, removes it from the `chunkBuffer`, and updates `nextExpectedChunkIndex` and `expectedIndex`. This process continues until there are no more consecutive chunks in the `chunkBuffer`.

[0164] `createTempFile()`: Creates a uniquely named temporary file in the Flink Task Manager's local file system. Standard Java file operations can be used. The filename typically includes the original filename and UUID to avoid conflicts.

[0165] uploadLocalFileToMinio():

[0166] Open a local file: Use the FileSystem API to open an input stream for a local temporary file.

[0167] MinIO Upload: Use the minioClient.putObject() method to send the local file's input stream, filename (as the object name in MinIO), file size, and other information to the target MinIO Bucket.

[0168] Retry mechanism: If a network error or other recoverable anomaly occurs during the upload process, a limited number of retries will be made (e.g., 3 times), with a short delay (backoff strategy) added between each retry.

[0169] Clean up local files: Once the upload is successful, delete local temporary files to free up disk space.

[0170] cleanupState(): This method is called after all chunks of a file have been processed and uploaded. It clears all Flink states associated with the current key (filename) (nextExpectedChunkIndex, chunkBuffer, isFileComplete, tempFilePathState), freeing up memory and preparing for processing the next new file.

[0171] Figure 4 The figure shown is an unstructured file transfer system based on Flink proposed in this application. The system includes:

[0172] The data object generation module is used to read unstructured files from object storage using a custom Source operator, encapsulate the file content into a data object containing the filename and a byte array, and output it downstream.

[0173] The data segmentation module is used to segment the data object into several data blocks using the flatMap operator. Each data block contains a file name, block number, byte data, and end block marker, forming a data block stream.

[0174] The group routing module is used to group the data block stream by keyBy based on the file name, so that all data blocks of the same file are routed to the same parallel task instance.

[0175] The ordered processing module is used to perform ordered processing on the grouped data blocks using the KeyedProcessFunction operator and transmit them to the target end. Specifically, it includes:

[0176] An initialization unit is used to create a connection with the target object storage and initialize state variables, including the expected block sequence number, out-of-order block cache, file completion flag, and temporary file path.

[0177] The processing unit is used to receive data blocks. When the current block sequence number is equal to the expected block sequence number, the current data block is written to the local temporary file and the expected block sequence number is updated. Otherwise, the current data block is stored in the out-of-order block cache.

[0178] The cyclic checking unit is used to check whether there are consecutive data blocks in the out-of-order block cache. If they exist, they are written to the local temporary file and the expected block sequence number is updated.

[0179] The completion unit is used to upload the local temporary file to the target object storage and clean up the relevant state variables after receiving the last block marker and completing the sequential writing of all data blocks.

[0180] The above system implements the steps of the Flink-based unstructured file transfer method disclosed in this application during runtime.

[0181] The flowcharts and block diagrams in the accompanying drawings illustrate possible implementations of systems, methods, and computer program products according to various embodiments of this application, including architecture, functionality, and operation. In these figures, each block may represent a module, program segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should be noted that each block in the block diagrams and / or flowcharts, and combinations thereof, can be implemented using either a dedicated hardware-based system or a combination of dedicated hardware and computer instructions to achieve the specified function or operation.

[0182] like Figure 5 As shown in the illustration, an embodiment of this application also discloses an electronic device, including: a processor 310, a communication interface 320, a memory 330 for storing a processor-executable computer program, and a communication bus 340. The processor 310, communication interface 320, and memory 330 communicate with each other via the communication bus 340. The processor 310 executes the executable computer program to implement the steps of the aforementioned Flink-based unstructured file transfer method.

[0183] It is understood that, in addition to memory and a processor, this electronic device may also include input devices (such as a keyboard), output devices (such as a display), and other communication modules. These input devices, output devices, and other communication modules all communicate with the processor through I / O interfaces (i.e., input / output interfaces).

[0184] The operations described in this application can be implemented by writing computer program code using one or more programming languages ​​or a combination thereof. The programming languages ​​include, but are not limited to, the following types:

[0185] Object-oriented programming languages, such as Java, Smalltalk, C++, etc.

[0186] Conventional procedural programming languages, such as "C" or similar programming languages.

[0187] The execution methods of program code include, but are not limited to:

[0188] It runs entirely on the user's computer;

[0189] Part of it executes on the user's computer, and part of it executes on a remote computer;

[0190] Execute as a standalone software package;

[0191] It is executed entirely on a remote computer or server.

[0192] In scenarios involving remote computers, the remote computer can connect to the user's computer via any type of network, including but not limited to local area networks (LANs) or wide area networks (WANs). Furthermore, the remote computer can also connect to external computers through an internet service provider, for example, by utilizing the internet for connection.

[0193] Furthermore, this application also discloses a computer-readable storage medium, wherein when the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is able to perform the various steps of the Flink-based unstructured file transfer method disclosed in this application.

[0194] In the context of this application, a computer-readable storage medium refers to a tangible medium capable of storing computer program code and related data. Specific examples include, but are not limited to, the following:

[0195] (1) Portable computer disk: such as floppy disks and other removable magnetic storage media.

[0196] (2) Hard disk: including mechanical hard disks and solid-state hard disks and other fixed storage devices.

[0197] (3) Random Access Memory (RAM): A volatile storage medium used for temporary storage of data and program code.

[0198] (4) Read-only memory (ROM): a non-volatile storage medium used to store fixed programs and data.

[0199] (5) Erasable programmable read-only memory (EPROM) or flash memory: non-volatile storage media that supports multiple erasures and reprogrammings.

[0200] (6) Fiber optic storage devices: storage media based on fiber optic technology.

[0201] (7) Portable compact disc read-only memory (CD-ROM): a read-only medium that stores data in the form of an optical disc.

[0202] (8) Optical storage devices: such as DVDs, Blu-ray discs and other storage media based on optical principles.

[0203] (9) Magnetic storage devices: such as magnetic tapes, disks and other storage media based on magnetic principles.

[0204] (10) Any suitable combination of the above: for example, combining multiple storage media to meet different storage needs.

[0205] These computer-readable storage media can be used to store the program code and related data described in this application to support program execution and persistent data storage.

[0206] Specifically, according to embodiments of this application, the processes described in the flowcharts can be implemented as computer software programs. For example, embodiments of this application relate to a computer program product comprising a computer program carried on a non-transitory computer-readable medium. This computer program contains program code for executing the Flink-based unstructured file transfer method disclosed in this application. When this computer program is executed by a processing system, it can achieve the functions defined in the embodiments of this application.

[0207] While the foregoing discussion contains several specific implementation details, these details should not be construed as limiting the scope of this application. The above description is merely a preferred embodiment of this application and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of this application is not limited to technical solutions formed by specific combinations of the above-described technical features. Furthermore, this application should also cover other technical solutions formed by any combination of the above-described technical features or their equivalents without departing from the foregoing disclosed concept.

[0208] Those skilled in the art should also understand that modifications can be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features, without departing from the spirit and scope of the technical solutions of the embodiments of this application. These modifications or substitutions will not cause the essence of the corresponding technical solutions to deviate from the core spirit and scope of the technical solutions of the embodiments of this application.< / string> < / boolean> < / long> < / string> < / boolean> < / long> < / string> < / boolean> < / long>

Claims

1. A method for transferring unstructured files based on Flink, characterized in that, The method includes: S1. Read unstructured files from object storage using a custom Source operator, encapsulate the file content into a data object containing the filename and a byte array, and output it downstream; S2. The data object is divided into several data blocks using the flatMap operator. Each data block contains a file name, block number, byte data, and end block marker, forming a data block stream. S3. Group the data block stream by keyBy based on the file name, so that all data blocks of the same file are routed to the same parallel task instance; S4. The KeyedProcessFunction operator is used to perform ordered processing on the grouped data blocks and transmit them to the target end. Specifically, this includes: S41. Initialization phase: Create a connection with the target object storage and initialize state variables, including expected block sequence number, out-of-order block cache, file completion flag and temporary file path; S42. Processing stage: Receive data blocks. When the current block sequence number is equal to the expected block sequence number, write the current data block to the local temporary file and update the expected block sequence number. Otherwise, store the current data block in the out-of-order block cache. S43, Cyclic Check Phase: Check if there are consecutive data blocks in the out-of-order block cache. If they exist, write them to the local temporary file and update the expected block sequence number. S44. Completion Phase: After receiving the last block marker and completing the sequential writing of all data blocks, upload the local temporary file to the target object storage and clean up the relevant state variables.

2. The method according to claim 1, characterized in that, The step S1, which involves reading unstructured files from object storage using a custom Source operator, specifically includes: Initialize the MinIO client, and configure the server endpoint, access key, and key. List objects under the source Bucket based on directory prefixes, filter directory objects, and then traverse the files; File byte data is read into memory through the input stream, encapsulated into a FileContent object, and output through the context collector.

3. The method according to claim 1, characterized in that, The data block mentioned in step S2 is a FileChunk object. The file is split by a preset block size, the block number is incremented from 0, and the last block is marked as IsLastChunk.

4. The method according to claim 1, characterized in that, Step S2, which involves dividing the data object into several data blocks using the flatMap operator, specifically includes: Extract filenames and file byte data, and then segment them cyclically according to a preset block size; Calculate the actual length of the current block and copy the corresponding bytes of data; Determine if it is the last block, generate a FileChunk object containing the filename, incrementing block number, byte data, and last block marker, and output it.

5. The method according to claim 1, characterized in that, The state variables mentioned in step S41 specifically include: nextExpectedChunkIndex:ValueState <long> Type, stores the sequence number of the next block to be received, initially set to 0;< / long> chunkBuffer:MapState<Long, byte[]> Type: Out-of-order block cache; key is block sequence number; value is block data. isFileComplete:ValueState <boolean> Type, indicates whether file reception is complete, initially false;< / boolean> tempFilePathState:ValueState <string> Type, path to store local temporary files.< / string> 6. The method according to claim 1, characterized in that, Step S42, which involves writing the current data block to a local temporary file, specifically includes: If the temporary file path status is empty, a unique temporary file name is generated and the status is updated; Open the local temporary file in append mode and write the current data block bytes of data.

7. The method according to claim 1, characterized in that, Step S43 includes: Loop through the out-of-order block cache to check if there is a data block with a sequence number equal to the current expected block sequence number; If it exists, write the data block to a local temporary file, remove it from the out-of-order block cache, update the expected block sequence number, and repeat the above process until there are no consecutive sequence numbers.

8. The method according to claim 1, characterized in that, The step S44, which involves uploading local temporary files to the target object storage, specifically includes: Get the local temporary file path and open the file input stream; The object storage SDK is invoked to perform an upload operation. After a successful upload, the local temporary file is deleted. The upload operation includes a retry mechanism, which performs a limited number of retries when a recoverable exception occurs.

9. The method according to claim 1, characterized in that, The method also includes a breakpoint resume mechanism: by recording the local temporary file path through the temporary file path, and combining the expected block sequence number and the out-of-order block cache state, the file merging and uploading can be resumed from the breakpoint after the Flink task restarts.

10. An unstructured file transfer system based on Flink, characterized in that, The system includes: The data object generation module is used to read unstructured files from object storage using a custom Source operator, encapsulate the file content into a data object containing the filename and a byte array, and output it downstream. The data segmentation module is used to segment the data object into several data blocks using the flatMap operator. Each data block contains a file name, block number, byte data, and end block marker, forming a data block stream. The group routing module is used to group the data block stream by keyBy based on the file name, so that all data blocks of the same file are routed to the same parallel task instance. The ordered processing module is used to perform ordered processing on the grouped data blocks using the KeyedProcessFunction operator and transmit them to the target end. Specifically, it includes: An initialization unit is used to create a connection with the target object storage and initialize state variables, including the expected block sequence number, out-of-order block cache, file completion flag, and temporary file path. The processing unit is used to receive data blocks. When the current block sequence number is equal to the expected block sequence number, the current data block is written to the local temporary file and the expected block sequence number is updated. Otherwise, the current data block is stored in the out-of-order block cache. The cyclic checking unit is used to check whether there are consecutive data blocks in the out-of-order block cache. If they exist, they are written to the local temporary file and the expected block sequence number is updated. The completion unit is used to upload the local temporary file to the target object storage and clean up the relevant state variables after receiving the last block marker and completing the sequential writing of all data blocks.

Images