Unstructured data synchronization method and system based on Flink

Abstract

The invention provides an unstructured data synchronization method and system based on Flink, and relates to the technical field of big data process.The method comprises the steps that fragmentation processing is conducted on unstructured data, hierarchical storage of the data in a heap memory layer and a RocksDB layer is achieved on the basis of access feature scores, data partition optimization is conducted in combination with deep reinforcement learning, and the unstructured data are synchronized. And the parallelism degree parameter and the resource quota are dynamically adjusted by calculating the back pressure index value, and the data synchronization reliability is ensured by adopting a check point snapshot mechanism, so that the data processing efficiency and the system throughput are effectively improved.

Description

Technical Field

[0001] The present invention relates to the technical field of big data processing, and particularly relates to an unstructured data synchronization method and system based on Flink. Background Art

[0002] With the development of big data technology, the management and synchronization of unstructured data have become an important part of enterprise data processing. Traditional data processing systems are mainly designed for structured data, and have limited processing capabilities for unstructured data such as images, videos, and documents; As a stream processing framework, Apache Flink provides the ability to process real-time data and can be used to build high-throughput and low-latency data processing pipelines. In the scenario of unstructured data synchronization, problems in multiple aspects such as data sharding, memory management, and performance optimization need to be solved. Existing unstructured data synchronization technologies based on Flink still adopt a unified storage strategy, unable to perform hierarchical storage optimization according to the heat characteristics of data, resulting in low access efficiency of hot data, cold data occupying valuable memory resources, low overall storage efficiency, lack of an effective dynamic load balancing mechanism, unable to adaptively adjust the processing strategy according to the real-time system load conditions, prone to backpressure phenomena, affecting the overall data processing throughput, and lack of an efficient state management and checkpoint mechanism. Once the system fails, it is difficult to quickly restore the data processing state, resulting in data consistency problems and increased processing delays, unable to meet the requirements of high-reliability data synchronization, etc.; Therefore, there is an urgent need for a solution to solve the problems existing in the prior art. Summary of the Invention

[0003] Embodiments of the present invention provide an unstructured data synchronization method and system based on Flink, which can at least solve some of the problems existing in the prior art.

[0004] In the first aspect of the embodiments of the present invention, an unstructured data synchronization method based on Flink is provided, including: Obtain the storage path of the unstructured data to be processed and perform sharding processing on the unstructured data to obtain sharded data blocks; Calculate a feature score based on the access time interval and access times of the sharded data blocks and generate a feature score table. Write the sharded data blocks with a feature score greater than a first threshold into the heap memory layer, and write other sharded data blocks into the RocksDB layer. Divide the sharded data blocks in the heap memory layer into multiple data partitions and generate an initial data partition table. Collect data access latency and calculate a latency prediction value in combination with deep reinforcement learning. Perform gradient iteration to generate a first optimization threshold and trigger data migration to obtain a data partition table; Read the unstructured data based on the preset buffer size according to the data partition table, and construct the read unstructured data into binary data of the metadata area and the data payload area; Calculate the ratio of the data volume of the binary data to the processing time limit as the processing load coefficient and divide it into micro-batches. Calculate the ratio of the data processing rate to the backlog data volume as the backpressure index value. When the backpressure index value exceeds the preset second threshold, perform a population iteration operation to obtain the parallelism parameter and resource quota corresponding to the maximum throughput and update them to the processing operator. Allocate time slices according to weighted round-robin, generate a micro-batch scheduling sequence, and perform data processing on the micro-batches; Save the checkpoint snapshot, and restore the data synchronization state according to the checkpoint snapshot in case of a failure.

[0005] In an alternative embodiment, Obtain the storage path of the unstructured data to be processed and perform sharding processing on the unstructured data to obtain sharded data blocks, including: Obtain the storage path of the unstructured data using the file system interface, where the storage path includes the data storage location and the data file name; Shard the unstructured data according to the preset maximum shard size, sequentially traverse the unstructured data and divide the data into blocks according to the byte size to generate multiple sharded data blocks; Calculate the data size and index position of each sharded data block, and record the data size and index position in the shard index table.

[0006] In an alternative embodiment, Calculate the feature score based on the access time interval and access times of the sharded data blocks and generate a feature score table. Write the sharded data blocks with feature scores greater than the first threshold into the heap memory layer, and write the other sharded data blocks into the RocksDB layer. Divide the sharded data blocks in the heap memory layer into multiple data partitions and generate an initial data partition table, including: Extract the access time interval and access times of the sharded data blocks. Take the time difference between two adjacent accesses as the access time interval, and the number of requests for the sharded data block within the unit time as the access times. Add the product of the reciprocal of the access time interval and the preset first weight coefficient to the product of the access times and the preset second weight coefficient to obtain the feature score. The sum of the first weight coefficient and the second weight coefficient is 1. Combine the identifier, access time interval, access times, feature score, and current storage location identifier corresponding to the sharded data block to construct the feature score table; Calculate the average value of all feature scores in the feature score table, add the product of the average value and the smoothing factor to the product of the previous cycle threshold obtained in advance and the complementary value of the smoothing factor to obtain the first threshold. Subtract the smoothing factor from 1 to obtain the complementary value of the smoothing factor. Write the sharded data blocks with feature scores greater than the first threshold into the heap memory layer, and write the sharded data blocks with feature scores less than or equal to the first threshold into the RocksDB layer; Group the sharded data blocks in the heap memory layer according to a fixed partition size, assign a unique identifier to each data partition, and write the partition number, start position offset, number of sharded data blocks, total partition data size, and current partition status of the data partition into the initial data partition table.

[0007] In an alternative embodiment, Collect data access latency and calculate the latency prediction value in combination with deep reinforcement learning. Perform gradient iteration to generate the first optimization threshold and trigger data migration to obtain the data partition table, including: Collect data access latency to obtain response time data. Extract the average latency, latency variance, and access frequency fluctuation degree in the response time data as the latency feature vector, and perform normalization processing on the latency feature vector to generate the latency feature matrix; Input the latency feature matrix, the current system load level, and the storage capacity utilization rate into the deep reinforcement learning network, extract the temporal features of data access to obtain the long-term dependence relationship, and determine the migration rules of sharded data blocks between different storage layers based on the long-term dependence relationship; Use a sliding time window to segment the historical latency sequence, perform iterative training on the latency data within each time window to obtain the latency prediction value, perform gradient iteration to update the training parameters, calculate the dynamic adjustment factor based on the current system load level and storage capacity utilization rate, and add the product of the latency prediction value and the dynamic adjustment factor to the product of the first threshold of the previous iteration cycle and the complementary value of the dynamic adjustment factor to obtain the first optimization threshold; Calculate the geometric mean of the access latency change amount, access frequency, and storage overhead of the sharded data blocks as the migration priority score and generate a migration queue. Identify the sharded data blocks to be migrated from the migration queue according to the first optimization threshold, perform data migration on the sharded data blocks to be migrated, and update the initial data partition table based on the distribution information of the sharded data blocks after migration to obtain the data partition table.

[0008] In an alternative embodiment, According to the preset buffer size, read unstructured data based on the data partition table and construct the read unstructured data into binary data of the metadata area and the data payload area, including: Read unstructured data from the data partition table according to the preset buffer size, read the unstructured data in chunks and cache it in memory; Extract the file name, file size, file type, creation time, access permission, and storage location from the unstructured data to construct a metadata area, and construct the actual content information in the unstructured data as a data payload area; Add a delimiter identifier at the end of the metadata area, and splice the metadata area and the data payload area in sequence according to the preset format to generate binary data.

[0009] In an alternative embodiment, Calculate the ratio of the data volume of the binary data to the processing time limit as the processing load coefficient and divide it into micro-batches. Calculate the ratio of the data processing rate to the backlog data volume as the backpressure index value. When the backpressure index value exceeds a preset second threshold, perform a population iteration operation to obtain the parallelism parameter and resource quota corresponding to the maximum throughput and update them to the processing operator, including: Calculate the ratio of the data volume of the binary data to the processing time limit within a preset time window to obtain the processing load coefficient, determine the number of micro-batches based on the system resource capacity, and divide the processing load coefficient by the number of micro-batches to divide the micro-batches; Calculate the ratio of the data processing rate to the backlog data volume to obtain the backpressure index value. When the backpressure index value exceeds a preset second threshold, perform a population iteration, construct multiple population individuals including parallelism parameters and resource quotas, encode the parallelism parameters and resource quotas of each population individual, where the value range of the parallelism parameter is from the minimum core number of the system to the maximum core number of the system, and the value range of the resource quota is from the minimum resource limit of the system to the maximum resource limit of the system; Calculate the system throughput corresponding to each population individual as the individual fitness value, select high-quality individuals from the population individuals through the roulette wheel algorithm, perform crossover operations and mutation operations on the parallelism parameters and resource quotas of the high-quality individuals to generate new population individuals, and retain the individual with the highest fitness in the population individuals as the optimal individual. Repeat the individual selection, crossover operation, and mutation operation until the fitness value of the population individuals converges or reaches the maximum number of iterations; Obtain the parallelism parameter and resource quota corresponding to the global optimal individual from the population individuals and update them to the processing operator.

[0010] In an alternative embodiment, Generate a micro-batch scheduling sequence according to weighted round-robin allocation of execution time slices and perform data processing on the micro-batches, including: Obtain the data volume of each micro - batch and the corresponding processing priority, calculate the micro - batch weight, multiply the micro - batch weight by the preset scheduling period to obtain the execution time slice of the micro - batch, and perform an initial allocation of the execution time slice using weighted round - robin; Obtain the processing throughput of each processing node in the historical processing records, calculate the difference between the historical processing time and the current time to obtain the time decay factor, multiply the processing throughput by the time decay factor and accumulate, and divide by the sum of the time decay factors to obtain the historical processing performance score corresponding to each processing node; Collect the number of CPU cores, memory capacity, and network bandwidth of the processing node to obtain the available resource vector, predict the resource demand vector based on the data scale and processing complexity of the micro - batch, and collect the ratio of the allocated resource amount to the total resource amount of the processing node to obtain the resource utilization rate; Divide the available resource vector by the resource demand vector and multiply by the resource utilization rate to obtain the resource matching degree; collect the average value of the CPU usage rate, memory usage rate, and network usage rate of the processing node to obtain the current load, subtract the average value of the loads of all processing nodes from the current load, divide by the maximum system load, take the absolute value, and subtract the calculation result from 1 to obtain the load balancing factor; Multiply the historical processing performance score, resource matching degree, and load balancing factor by the preset weight coefficients respectively and sum them to obtain the processing node affinity, and multiply the execution time slice by the processing node affinity to obtain the comprehensive scheduling score; Select the processing node with the maximum processing node affinity as the task migration target node, generate a micro - batch scheduling sequence based on the comprehensive scheduling score, and distribute the micro - batch data to the task migration target node, and perform data processing within the allocated execution time slice to obtain the processing result.

[0011] In the second aspect of the embodiments of the present invention, a Flink - based unstructured data synchronization system is provided, including: A first unit for obtaining the storage path of the unstructured data to be processed and performing sharding processing on the unstructured data to obtain sharded data blocks; A second unit for calculating feature scores based on the access time interval and access times of the sharded data blocks and generating a feature score table, writing the sharded data blocks with feature scores greater than the first threshold into the heap memory layer, writing other sharded data blocks into the RocksDB layer, dividing the sharded data blocks in the heap memory layer into multiple data partitions and generating an initial data partition table, collecting data access latency and calculating a latency prediction value in combination with deep reinforcement learning, performing gradient iteration to generate a first optimization threshold and triggering data migration to obtain a data partition table; A third unit for reading unstructured data based on the data partition table according to the preset buffer size and constructing the read unstructured data into binary data of a metadata area and a data payload area; A fourth unit is configured to calculate the ratio of the data volume of the binary data to the processing time limit as a processing load factor and divide micro-batches, calculate the ratio of the data processing rate to the backlog data volume as a backpressure index value, and when the backpressure index value exceeds a preset second threshold, perform a population iteration operation to obtain a parallelism parameter and a resource quota corresponding to the maximum throughput and update them to a processing operator, allocate time slices according to weighted round-robin, generate a micro-batch scheduling sequence, and perform data processing on the micro-batches; A fifth unit is configured to save a checkpoint snapshot and restore the data synchronization state according to the checkpoint snapshot in case of a failure.

[0012] In a third aspect of the embodiments of the present invention, an electronic device is provided, including: A processor and a memory for storing processor-executable instructions, wherein the processor is configured to call the instructions stored in the memory to execute the method described above.

[0013] In a fourth aspect of the embodiments of the present invention, a computer-readable storage medium is provided, on which computer program instructions are stored, and when the computer program instructions are executed by a processor, the method described above is implemented.

[0014] In the present invention, by performing sharding processing on unstructured data and performing hierarchical storage management based on access characteristics, the data access efficiency is improved, the data retrieval time is effectively reduced, and the overall system performance is improved. By using a deep reinforcement learning method to predict the data access latency and dynamically optimize the data partitioning strategy, combined with the binary data construction method, not only the resource consumption in the data transmission and processing process is reduced, but also the flexibility and efficiency of data processing are improved. By calculating the processing load factor and the backpressure index value, micro-batch division and dynamic resource scheduling are realized, and the checkpoint snapshot mechanism is used to ensure the stable operation and fault recovery ability of the system under high load, significantly improving the throughput and reliability in the large-scale unstructured data synchronization process. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 It is a flowchart of the method for unstructured data synchronization based on Flink according to the embodiments of the present invention; Figure 2 It is a flowchart of the process of performing a population iteration operation to obtain a parallelism parameter and a resource quota corresponding to the maximum throughput. DETAILED DESCRIPTION

[0016] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part rather than all of the embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0017] The technical solutions of the present invention will be described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments.

[0018] Figure 1 The following is a schematic flowchart of the method for unstructured data synchronization based on Flink in the embodiments of the present invention. As Figure 1 shown, the method includes: Obtain the storage path of the unstructured data to be processed and perform sharding processing on the unstructured data to obtain sharded data blocks; Calculate the feature scores based on the access time intervals and access counts of the sharded data blocks and generate a feature score table. Write the sharded data blocks with feature scores greater than the first threshold into the heap memory layer, and write the other sharded data blocks into the RocksDB layer. Divide the sharded data blocks in the heap memory layer into multiple data partitions and generate an initial data partition table. Collect the data access latency and calculate the latency prediction value in combination with deep reinforcement learning. Perform gradient iteration to generate the first optimization threshold and trigger data migration to obtain the data partition table; According to the preset buffer size, read the unstructured data based on the data partition table and construct the read unstructured data into binary data of a metadata area and a data payload area; Calculate the ratio of the data volume of the binary data to the processing time limit as the processing load coefficient and divide it into micro-batches. Calculate the ratio of the data processing rate to the backlogged data volume as the backpressure index value. When the backpressure index value exceeds the pre-set second threshold, perform a population iteration operation to obtain the parallelism parameter and resource quota corresponding to the maximum throughput and update them to the processing operator. Allocate time slices according to weighted round-robin, generate a micro-batch scheduling sequence, and perform data processing on the micro-batches; Save the checkpoint snapshot, and restore the data synchronization state according to the checkpoint snapshot in case of a failure.

[0019] In an alternative embodiment, Obtaining the storage path of the unstructured data to be processed and performing sharding processing on the unstructured data to obtain sharded data blocks includes: Obtain the storage path of unstructured data through the file system interface. The storage path includes the data storage location and the data file name; Fragment the unstructured data according to the preset maximum fragment size. Sequentially traverse the unstructured data and divide the data into blocks according to the byte size to generate multiple fragmented data blocks; Calculate the data size and index position of each fragmented data block, and record the data size and index position in the fragment index table.

[0020] Obtain the complete storage path information of the unstructured data through the file system interface. The storage path consists of two key components: the data storage location and the data file name. The data storage location indicates the location of the unstructured data in the storage system, such as " / data / documents / "; the data file name identifies the specific data file, such as "large_text_file.txt". Use standard file system APIs such as the open() and fopen() functions to obtain the path to ensure that the subsequent processing can accurately locate the target data. For example, the complete path " / data / documents / large_text_file.txt" can be obtained by calling the system function. In actual application scenarios, it is necessary to process various types of files in multiple directories, such as unstructured data like text documents, pictures, and videos. Use a recursive method to traverse the directory tree to obtain the file paths that meet the conditions.

[0021] After obtaining the storage path, fragment the unstructured data according to the preset maximum fragment size. The preset maximum fragment size is a parameter preset by the system according to factors such as processing capacity and memory limit, usually in bytes, such as 4MB, 8MB, or 16MB. Open the target unstructured data file and read the file content in a sequential traversal manner. During the traversal, divide the entire unstructured data file into multiple consecutive but non-overlapping data blocks according to the set maximum fragment size. For example, if the maximum fragment size is set to 4MB (i.e., 4,194,304 bytes), for a 10MB text file, it will be divided into 3 fragmented data blocks: the first two fragments are each 4MB, and the last fragment is 2MB.

[0022] Through a file pointer or streaming reading method, read a fixed-size data block each time. When the amount of data read reaches the preset maximum fragment size or reaches the end of the file, process the currently read content as a complete fragmented data block. To ensure data integrity, perform appropriate processing at the fragment boundary. For example, for a text file, it is necessary to ensure that a complete word or sentence is not split into different fragments, and select an appropriate split point, such as the end of a line or the boundary of a paragraph.

[0023] After generating the chunk data blocks for each sharded data block, calculate the data size and index position of each sharded data block and record them in the shard index table. The data size refers to the actual byte size of the shard, and the index position refers to the starting position of the shard in the original unstructured data. The shard index table is a data structure used to record and manage the metadata information of all shards, facilitating subsequent data access and processing.

[0024] For the example of the above 10MB file, the shard index table may contain the following information: Shard 1: Data size = 4,194,304 bytes, index position = 0; Shard 2: Data size = 4,194,304 bytes, index position = 4,194,304; Shard 3: Data size = 2,097,152 bytes, index position = 8,388,608; In the actual implementation process, the shard index table can be implemented using various data structures, such as arrays, linked lists, or hash tables. Select a suitable data structure according to specific requirements. To improve the retrieval efficiency of the index table, additional index fields, such as shard ID, creation timestamp, etc., are added to the index table. In addition to recording the size and position information of the shards, the shard index table can also contain other metadata that helps with data management, such as data type, data digest, or checksum, etc.

[0025] In this embodiment, large-scale unstructured data can be efficiently sharded, converting the originally difficult-to-process large data file into multiple data blocks suitable for parallel processing, while maintaining data integrity and indexability. This not only improves data processing efficiency but also lays a foundation for subsequent parallel computing, distributed storage, and fast retrieval, and has important value for big data processing systems.

[0026] In an alternative embodiment, Calculate the feature score based on the access time interval and access count of the sharded data block and generate a feature score table. Write the sharded data blocks with a feature score greater than the first threshold into the heap memory layer, and write other sharded data blocks into the RocksDB layer. Divide the sharded data blocks in the heap memory layer into multiple data partitions and generate an initial data partition table, including: Extract the access time interval and access count of the sharded data block. Take the time difference between two adjacent accesses as the access time interval, and the request count of the sharded data block within a unit time as the access count. Add the product of the reciprocal of the access time interval and a pre-set first weight coefficient to the product of the access count and a pre-set second weight coefficient to obtain a feature score. The sum of the first weight coefficient and the second weight coefficient is 1. Combine the identifier corresponding to the sharded data block, the access time interval, the access count, the feature score, and the current storage location identifier to construct a feature score table. Calculate the average value of all feature scores in the feature score table. Add the product of the average value and a smoothing factor to the product of a previously obtained previous cycle threshold and the complementary value of the smoothing factor to obtain a first threshold. Subtract the smoothing factor from 1 to get the complementary value of the smoothing factor. Write the sharded data blocks with feature scores greater than the first threshold into the heap memory layer, and write the sharded data blocks with feature scores less than or equal to the first threshold into the RocksDB layer. Group the sharded data blocks in the heap memory layer according to a fixed partition size, assign a unique identifier to each data partition, and write the partition number of the data partition, the starting position offset, the number of sharded data blocks, the total size of the partition data, and the current state of the partition into the initial data partition table.

[0027] Continuously monitor and record the access situation of each sharded data block. For a specific sharded data block, record the timestamp of each access. When enough access records are collected, start to extract the access time interval and access count of the sharded data block. The access time interval is obtained by calculating the time difference between two adjacent accesses. For example, if a data block is accessed at 10:00:00 and 10:00:30 respectively, then one of the access time intervals is 30 seconds. For the access count, the system will count the total number of times the data block is requested within a certain time window (such as 1 minute, 5 minutes, or 10 minutes).

[0028] After obtaining the data, calculate the feature score. The calculation formula for the feature score is: the product of the reciprocal of the access time interval and the first weight coefficient, plus the product of the access count and the second weight coefficient. In practical applications, the first weight coefficient can be set to 0.4, and the second weight coefficient can be set to 0.6 to ensure that their sum is 1. For example, for a sharded data block, if its average access time interval is 2 seconds and the access count within the specified time window is 50 times, then its feature score is: (1 / 2)*0.4 + 50*0.6 = 0.2 + 30 = 30.2.

[0029] To build a feature score table, the identifier of each shard data block (such as a unique ID), access time interval, access count, calculated feature score, and the current storage location identifier (such as "heap memory" or "RocksDB") are combined. Such a feature score table may contain thousands of records, with each record corresponding to the complete characteristic information of a shard data block.

[0030] In determining the first threshold, a dynamic adjustment mechanism is adopted. First, the average value of all feature scores in the feature score table is calculated. Suppose the average feature score in the current period is 25, the smoothing factor is set to 0.3, and the threshold in the previous period is 20. Then the first threshold in the current period is calculated as: 25 * 0.3 + 20 * (1 - 0.3) = 7.5 + 14 = 21.5, enabling the threshold to be smoothly adjusted as the data access pattern changes and avoiding performance issues caused by improper threshold setting.

[0031] Based on the calculated first threshold, the storage location of each shard data block is determined. For shard data blocks with a feature score exceeding the first threshold, they are written to the heap memory layer; while shard data blocks with a feature score less than or equal to the first threshold are written to the RocksDB layer. For example, for the above case where the first threshold is 21.5, the shard data block with a feature score of 30.2 will be written to the heap memory layer, and the shard data block with a feature score of 15.8 will be written to the RocksDB layer.

[0032] For the shard data blocks written to the heap memory layer, further data partition management is carried out. The shard data blocks in the heap memory layer are grouped according to a preset fixed partition size (such as each partition contains at most 100 shard data blocks or the total size does not exceed 10MB). Each data partition is assigned a unique identifier, such as consecutive integers starting from 1.

[0033] When building the initial data partition table, the following information is recorded for each data partition: partition number (such as 1, 2, 3, etc.), starting position offset (such as the starting address in memory), the number of shard data blocks contained (such as 87), the total size of the partition data (such as 8.7MB), and the current state of the partition (such as "active" or "full"). This information is organized into the initial data partition table to provide support for subsequent data access and management.

[0034] Intelligently store hot data in high-speed heap memory and cold data in RocksDB according to the access characteristics of data blocks, thus optimizing memory usage while ensuring performance. For example, in a practical application scenario, a total of 10,000 sharded data blocks are managed. After feature scoring calculation and threshold comparison, 2,000 data blocks are determined to be hot data and stored in heap memory, which are divided into 25 data partitions for management. The remaining 8,000 data blocks are stored in RocksDB as cold data to save memory resources. When the data access pattern changes, the distribution of data between different storage layers is dynamically adjusted by periodically recalculating the feature scores and updating the thresholds.

[0035] In this embodiment, through the dynamic evaluation mechanism of access time interval and access frequency, the access heat characteristics of sharded data blocks are accurately characterized, and the dynamic threshold for hierarchical storage is calculated in combination with the historical cycle threshold, avoiding the inflexibility brought by fixed thresholds, and being able to adjust the storage location in a timely manner according to the change of data access pattern. Through the feature-score-driven hierarchical storage mechanism, hot data with high-frequency access is stored in the heap memory layer, and cold data with low-frequency access is stored in the RocksDB layer, achieving an optimal balance between access efficiency and storage cost. The data in the heap memory layer is managed in fixed-size partitions, and the complete partition metadata information is maintained, which is convenient for subsequent efficient organization and management of memory data, improving the maintainability and scalability of the system.

[0036] In an alternative embodiment, Collect data access latency and calculate the latency prediction value in combination with deep reinforcement learning, perform gradient iteration to generate the first optimization threshold and trigger data migration to obtain a data partition table, including: Collect data access latency to obtain response time data, extract the average latency, latency variance, and access frequency fluctuation degree in the response time data as a latency feature vector, and perform normalization processing on the latency feature vector to generate a latency feature matrix; Input the latency feature matrix, the current system load level, and the storage capacity utilization rate into the deep reinforcement learning network, extract the temporal features of data access to obtain long-term dependencies, and determine the migration rules of sharded data blocks between different storage layers based on the long-term dependencies; Use a sliding time window to segment and process the historical latency sequence, perform iterative training on the latency data within each time window to obtain the latency prediction value, perform gradient iteration to update the training parameters, calculate the dynamic adjustment factor based on the current system load level and storage capacity utilization rate, and add the product of the latency prediction value and the dynamic adjustment factor to the product of the first threshold in the previous iteration cycle and the complementary value of the dynamic adjustment factor to obtain the first optimization threshold; Calculate the geometric mean of the access latency variation, access frequency, and storage overhead of the sharded data blocks as the migration priority score and generate a migration queue. Identify the sharded data blocks to be migrated from the migration queue according to the first optimization threshold, perform data migration on the sharded data blocks to be migrated, and update the initial data partition table based on the distribution information of the sharded data blocks after migration to obtain the data partition table.

[0037] Collect data access latency information through the system logger, including the time interval from the request start to the response completion for each data block, sample once every 5 minutes, and collect the response time data of the most recent 100 access operations. From the original latency data, calculate the average latency value, latency standard deviation, and access frequency fluctuation. For example, for a certain data shard A, its average access latency is calculated to be 15 milliseconds, the latency variance is 5 millisecond squared, and the access frequency fluctuation is 0.35. Combine to form a latency feature vector [15, 5, 0.35]. To eliminate the dimension difference, use the extreme value normalization method to process the feature vector, map the feature values of each dimension to the [0, 1] interval, and obtain the normalized feature vector [0.75, 0.25, 0.35]. After performing the same processing on all data shards, generate an overall latency feature matrix.

[0038] After the feature matrix is generated, input it together with the current system load level (such as CPU utilization rate of 80%) and storage layer capacity utilization rate (such as high-speed storage layer usage rate of 90%, medium-speed storage layer usage rate of 65%) into the deep reinforcement learning network. The deep reinforcement learning network adopts a long short-term memory (LSTM) cell structure, including an input layer, two hidden layers, and an output layer. The input layer receives the latency feature matrix and system state information. The first hidden layer contains 64 neurons, and the second hidden layer contains 32 neurons. The network extracts the temporal features of the data access pattern and identifies long-term dependencies, such as the difference in access patterns between weekdays and weekends, and the load change rules between day and night. For example, it is found that data shard B has a high access frequency and is latency-sensitive from 9 am to 11 am every day, and has less access at other times. Based on the identified long-term dependencies, determine the migration rules for sharded data blocks between different storage layers: for example, data blocks that are frequently accessed and latency-sensitive are preferentially migrated to the high-speed storage layer; data blocks with periodic changes in access frequency are migrated to the high-speed storage layer before the access peak and back to the low-speed layer during the low period.

[0039] To achieve accurate delay prediction, a sliding time window technique is adopted to process historical delay data. The window size is set to 24 hours, and the step size is 1 hour. Each window contains the delay data points of the most recent 24 hours. For each sliding window, the stochastic gradient descent algorithm is applied to perform iterative training, and the initial learning rate is set to 0.01. During the training process, when the validation error does not decrease for 5 consecutive iterations, the learning rate is halved. After 100 rounds of iterative training, the predicted access delay for the next time period of data shard C is 12 milliseconds. The dynamic adjustment factor α is calculated based on the current load level and storage capacity status. For example, when the utilization rate of the high-speed storage layer exceeds 85%, the α value is set to 0.7; when the system load exceeds 75%, the α value is further reduced to 0.6. Assuming the first threshold of the previous iteration period is 20 milliseconds, then the first optimized threshold for this round is calculated as: 12×0.6 + 20×(1 - 0.6) = 15.2 milliseconds. This threshold will be used as the key basis for determining whether a data block needs to be migrated.

[0040] Calculate the migration priorities of each shard data block. Taking data shard D as an example, the change in its access delay is 5 milliseconds (increasing from 15 milliseconds to 20 milliseconds), the access frequency is 10 times per second, and the storage overhead is 200 MB. Calculate the geometric mean of these three metrics to obtain a migration priority score of (5×10×200)^(1 / 3) ≈ 17.1. The same calculation is performed for all shard data blocks, and a migration queue is generated by sorting them in descending order of the priority scores, such as [D, E, F, G, H]. Based on the first optimized threshold of 15.2 milliseconds, the system identifies data blocks D and E in the queue with priority scores greater than this threshold as the data blocks to be migrated.

[0041] For the data blocks to be migrated, start the migration executor to copy data shard D from the low-speed storage layer to the high-speed storage layer, and update the metadata information after completion; also migrate data shard E to the corresponding target storage layer. The batch copy strategy is adopted during the migration process, with 4 MB of data in each batch to reduce the impact on system performance. After the migration is completed, update the initial data partition table to record the latest shard location information. For example, the new location of data shard D is "high-speed storage layer - node 3", and the new location of data shard E is "medium-speed storage layer - node 2". The updated data partition table will serve as the basis for the next round of optimization and adjustment.

[0042] In this embodiment, the average delay, delay variance, and access frequency fluctuation degree in the response time data are extracted to construct a delay feature vector, and normalization processing is performed, comprehensively characterizing the delay characteristics of data access, providing a standardized data basis for subsequent analysis. The delay feature matrix is combined with system load and storage utilization and input into a deep reinforcement learning network to achieve automatic learning of the long-term dependence relationship of data access patterns, enabling data migration decisions to adapt to complex access pattern changes. A sliding time window is used to segment and iteratively train the historical delay sequence, and the training parameters are continuously optimized through gradient iteration, improving the accuracy of delay prediction. The migration priority is calculated based on the geometric mean of the access delay change amount, access frequency, and storage overhead, ensuring the comprehensiveness and rationality of the migration decision.

[0043] In an alternative embodiment, Reading unstructured data based on a data partition table according to a preset buffer size and constructing the read unstructured data into binary data for a metadata area and a data payload area includes: Reading unstructured data from the data partition table according to the preset buffer size, reading the unstructured data in chunks and caching it in memory; Extracting the file name, file size, file type, creation time, access permission, and storage location from the unstructured data to construct a metadata area, and constructing the actual content information in the unstructured data into a data payload area; Adding a delimiter identifier at the end of the metadata area, and concatenating the metadata area and the data payload area in a preset format sequence to generate binary data.

[0044] Read unstructured data from the data partition table according to the preset buffer size. In practical applications, the preset buffer size is set to 8MB, which can usually achieve a good balance between memory usage and reading efficiency. Check the total amount of unstructured data to be read. For example, the size of a certain document file is 20MB, which exceeds the preset 8MB buffer size, and it is read in multiple blocks. Specifically, create an 8MB memory buffer, start from the starting position of the data partition table, and read 8MB of data into the buffer each time. For this 20MB document file, three read operations are performed: the first two times, 8MB of data is read each time, and the last time, the remaining 4MB of data is read. This mechanism of reading in chunks ensures that the system can process unstructured data of any size without causing a memory overflow due to loading too much data at one time.

[0045] After the reading is completed, extract metadata information from the unstructured data for constructing the metadata area. The metadata information includes key attributes such as file name, file size, file type, creation time, access permission, and storage location.

[0046] Extract the actual content information of the unstructured data and construct it into a data payload area. For document files, the actual content is the binary data of the document; for image files, the actual content is the binary data of the image; for video files, the actual content is the binary data of the video. Do not modify or convert this actual content, but retain the binary form as it is to ensure the integrity and accuracy of the data.

[0047] After constructing the metadata area, add a delimiter at the end of the metadata area to accurately distinguish the metadata area and the data payload area in subsequent processing. This delimiter can be a special string, such as "##METADATA_END##", or a special binary sequence, such as 8 consecutive bytes with all values being 0xFF. The length and content of the delimiter can be customized according to the specific application scenario, but it must be ensured that the delimiter does not appear in the normal content of the metadata area or the data payload area to avoid ambiguity.

[0048] Concatenate the metadata area and the data payload area in the preset format order to generate the final binary data. The preset format order is usually the metadata area first and then the data payload area, which is convenient for the receiving party to first parse the metadata to understand the basic information of the file and then process the data payload as needed. The concatenation process is to write the binary representation of the metadata area, the delimiter, and the binary data of the data payload area into a continuous memory block or file in order. For example, assume that the size of the JSON serialized metadata area is 500 bytes, the delimiter is "##METADATA_END##" (15 bytes), and the size of the data payload area is 20971520 bytes (20MB), then the total size of the generated binary data is 500 + 15 + 20971520 = 20972035 bytes.

[0049] In this embodiment, by performing block reading and caching through a preset buffer size, the risk of memory overflow caused by loading a large amount of data at one time is avoided, and the efficient batch processing of unstructured data is realized. The unstructured data is separated into a metadata area and a data payload area, and the basic attribute information of the file is reasonably extracted and organized, making the data structure clearer and facilitating subsequent retrieval and management. Adding a delimiter at the end of the metadata area and concatenating it in the preset format ensures the integrity and parsability of the binary data, and at the same time provides a boundary recognition mechanism for the metadata and the data payload, facilitating the rapid positioning and parsing of the data.

[0050] In an alternative embodiment, Calculate the ratio of the amount of binary data to the processing time limit as the processing load coefficient and divide it into micro-batches. Calculate the ratio of the data processing rate to the backlog data volume as the backpressure index value. When the backpressure index value exceeds a pre-set second threshold, perform a population iteration operation to obtain the parallelism parameter and resource quota corresponding to the maximum throughput and update them to the processing operator, including: Calculate the ratio of the amount of binary data to the processing time limit within a preset time window to obtain the processing load coefficient. Determine the number of micro-batches based on the system resource capacity and divide the micro-batches by dividing the processing load coefficient by the number of micro-batches; Calculate the ratio of the data processing rate to the backlog data volume to obtain the backpressure index value. When the backpressure index value exceeds a pre-set second threshold, perform a population iteration. Construct multiple population individuals containing parallelism parameters and resource quotas, and encode the parallelism parameters and resource quotas of each population individual. The value range of the parallelism parameter is from the minimum core number of the system to the maximum core number of the system, and the value range of the resource quota is from the minimum resource limit of the system to the maximum resource limit of the system; Calculate the system throughput corresponding to each population individual as the individual fitness value. Select high-quality individuals from the population individuals through the roulette wheel algorithm, perform crossover operations and mutation operations on the parallelism parameters and resource quotas of the high-quality individuals to generate new population individuals, and retain the individual with the highest fitness in the population individuals as the optimal individual. Repeat the individual selection, crossover operation, and mutation operation until the fitness value of the population individuals converges or reaches the maximum number of iterations; Obtain the parallelism parameter and resource quota corresponding to the global optimal individual from the population individuals and update them to the processing operator.

[0051] Figure 2 It is a schematic diagram of the process for performing a population iteration operation to obtain the parallelism parameter and resource quota corresponding to the maximum throughput. Perform processing load calculation and micro-batch division on the received binary data, and calculate the ratio of the amount of binary data to the processing time limit within a preset time window to obtain the processing load coefficient. For example, assume that the amount of data received within a 5-second time window is 500MB and the processing time limit is 2 seconds. Then the processing load coefficient is calculated as 500MB / 2 seconds = 250MB / second. Determine the number of micro-batches based on the system resource capacity. If the system available resource is an 8-core processor and each core can process 1 micro-batch, then the number of micro-batches is 8. Divide the processing load coefficient by the number of micro-batches to divide the micro-batches, that is, 250MB / second divided by 8 is equal to 31.25MB / second, indicating the amount of data that each micro-batch needs to process.

[0052] Continuously monitor the data processing situation, calculate the ratio of the data processing rate to the backlog data volume to obtain the backpressure index value. For example, if the current system data processing rate is 200 MB per second and the backlog data volume is 600 MB, then the backpressure index value is 200 MB per second divided by 600 MB, which is equal to 0.33 times per second. The second threshold is preset to 0.5 times per second. When the backpressure index value of 0.33 is less than the threshold of 0.5, it indicates that the processing speed is slower than the data backlog speed, and resource adaptive adjustment needs to be triggered.

[0053] When the backpressure index value exceeds the preset second threshold, start performing population iteration operations to construct multiple population individuals containing parallelism parameters and resource quotas. For example, create an initial population of 20 individuals, and each individual contains two parameters: the parallelism parameter and the resource quota. Encode the parallelism parameter and the resource quota of each population individual, using binary encoding. The value range of the parallelism parameter is from the minimum core number 4 of the system to the maximum core number 32 of the system, and the value range of the resource quota is from the minimum resource limit 2 GB of the system to the maximum resource limit 16 GB of the system.

[0054] For each population individual, calculate the corresponding system throughput as the individual fitness value. For example, for an individual with a parallelism of 16 and a resource quota of 8 GB, measure its actual throughput of processing data as 350 MB per second, then the fitness value of this individual is 350. Select high-quality individuals from the population individuals through the roulette wheel algorithm. During the roulette wheel selection process, the probability of each individual being selected is proportional to its fitness. If the total fitness in the current population is 5000 and the fitness of a certain individual is 350, then the probability of this individual being selected is 350 / 5000 = 7%.

[0055] Perform crossover operations and mutation operations on the selected high-quality individuals to generate new population individuals. The crossover operation exchanges part of the coding information of two parent individuals. For example, the parallelism of parent individual A is 16 (binary 10000), and the resource quota is 8 GB (binary 1000); the parallelism of parent individual B is 24 (binary 11000), and the resource quota is 12 GB (binary 1100). Select the crossover points as the third bit of the parallelism and the second bit of the resource quota. After crossover, two new individuals are obtained: the parallelism is 20 (binary 10100), the resource quota is 10 GB (binary 1010), and the parallelism is 28 (binary 11100), the resource quota is 14 GB (binary 1110).

[0056] The mutation operation randomly changes some bits in the individual encoding. For example, for an individual with a parallelism of 20 (binary 10100) and a resource quota of 10GB (binary 1010), the fourth bit of the parallelism and the third bit of the resource quota are randomly selected for mutation. After mutation, a new individual with a parallelism of 28 (binary 11100) and a resource quota of 14GB (binary 1110) is obtained. The individual with the highest fitness in the population is retained as the optimal individual. If the highest fitness in the current population is 380, and the corresponding individual has a parallelism of 20 and a resource quota of 10GB, then this individual is marked as the current optimal individual.

[0057] Repeat the execution of individual selection, crossover operation, and mutation operation until the fitness values of the population individuals converge or reach the maximum number of iterations. The convergence of fitness values means that the change in the optimal fitness for 10 consecutive generations is less than 1%; the maximum number of iterations is set to 100. For example, when the iteration reaches the 50th generation, the optimal fitness has fluctuated around 410 for 10 consecutive generations, and the change rate is less than 1%, then it is considered that the algorithm has converged and the iteration process is terminated.

[0058] After the iteration ends, obtain the parallelism parameter and resource quota corresponding to the global optimal individual from the population individuals. For example, the parallelism parameter of the global optimal individual is 24, and the resource quota is 12GB, and the corresponding system throughput is 412MB / second. Update these parameters to the processing operator to achieve adaptive adjustment of system resources. After receiving the new parallelism parameter and resource quota, the processing operator dynamically adjusts the number of parallel tasks and resource limits to achieve the optimal processing efficiency when processing the data stream.

[0059] In this embodiment, through the dynamic calculation of the processing load coefficient and the number of micro-batches, the adaptive segmentation of data processing tasks is realized, ensuring the processing load balance of each micro-batch, avoiding performance bottlenecks caused by uneven task allocation, introducing the backpressure index value to monitor the system pressure state, and when the system has insufficient processing capacity, triggering the population iteration optimization in a timely manner, realizing the dynamic adjustment of system resource configuration, improving the adaptive ability of the system, adopting a population iteration mechanism with two-dimensional encoding of parallelism parameters and resource quotas, exploring the optimal configuration combination within the scope of system resource constraints, and combining the roulette wheel selection algorithm, crossover operation, and mutation operation to enhance the population diversity and avoid falling into local optimal solutions.

[0060] In an alternative embodiment, According to the weighted round-robin allocation to execute time slices, generating a micro-batch scheduling sequence and performing data processing on the micro-batches includes: Obtain the data volume of the micro-batch and the processing priority corresponding to each micro-batch, calculate the micro-batch weight, multiply the micro-batch weight by the preset scheduling period to obtain the execution time slice of the micro-batch, and perform an initial allocation of the execution time slices using weighted round-robin. Obtain the processing throughput of each processing node in the historical processing record, calculate the difference between the historical processing time and the current time to obtain the time decay factor, multiply the processing throughput by the time decay factor and accumulate, and divide by the sum of the time decay factors to obtain the historical processing performance score corresponding to each processing node; Collect the number of CPU cores, memory capacity, and network bandwidth of the processing node to obtain an available resource vector, predict the resource demand vector based on the data scale and processing complexity of the micro-batch, and collect the ratio of the allocated resource amount to the total resource amount of the processing node to obtain the resource utilization rate; Divide the available resource vector by the resource demand vector and multiply by the resource utilization rate to obtain the resource matching degree; collect the average value of the CPU usage rate, memory usage rate, and network usage rate of the processing node to obtain the current load, subtract the average value of the loads of all processing nodes from the current load, divide by the maximum system load, and take the absolute value, and use 1 minus the calculation result to obtain the load balancing factor; Multiply the historical processing performance score, resource matching degree, and load balancing factor by the preset weight coefficients respectively and sum to obtain the processing node affinity, and multiply the execution time slice by the processing node affinity to obtain the comprehensive scheduling score; Select the processing node with the maximum processing node affinity as the task migration target node, generate a micro-batch scheduling sequence based on the comprehensive scheduling score, and distribute the micro-batch data to the task migration target node, and execute data processing within the allocated execution time slice to obtain the processing result.

[0061] Obtain the data volume of the micro-batch and the processing priority corresponding to each micro-batch. For example, in a data stream processing system, there are three micro-batches A, B, and C, with data volumes of 50MB, 30MB, and 80MB respectively, and processing priorities of 5, 3, and 4 (the priority range is 1-5, and the larger the value, the higher the priority). According to the product of the micro-batch data volume and the processing priority, calculate the micro-batch weight, and the weights of the three micro-batches are 250, 90, and 320 respectively. The preset scheduling period is 100ms, multiply the micro-batch weight by the preset scheduling period and perform normalization processing to obtain the execution time slices of the micro-batches, which are 37.9ms, 13.6ms, and 48.5ms respectively. Use the weighted round-robin algorithm to perform initial allocation on these execution time slices to ensure that micro-batches with high weights obtain more processing time.

[0062] When processing node selection, obtain the processing throughput of each processing node in the historical processing records. Assume there are two processing nodes N1 and N2 in the system. The processing throughputs of N1 in the recent five processing records are 120MB / s, 110MB / s, 125MB / s, 105MB / s, and 130MB / s respectively, and the processing times are 5 minutes ago, 4 minutes ago, 3 minutes ago, 2 minutes ago, and 1 minute ago; while the processing throughputs of N2 are 95MB / s, 105MB / s, 90MB / s, 100MB / s, and 110MB / s respectively, and the processing times are the same as those of N1. Calculate the difference between the historical processing time and the current time to obtain the time decay factor. Adopt a time decay function, such as an exponential decay function. The decay factors for the five processing records are 0.6, 0.7, 0.8, 0.9, and 1.0 respectively. Multiply the processing throughput by the time decay factor and accumulate them. The result for N1 is 457.5, and for N2 is 380.5. Then divide by the sum of the time decay factors, which is 4.0. Finally, the historical processing performance score of N1 is 114.38, and that of N2 is 95.13.

[0063] Collect the available resource situation of the processing nodes, including the number of CPU cores, memory capacity, and network bandwidth. Assume N1 has 16-core CPU, 64GB of memory, and 10Gbps network bandwidth, and N2 has 8-core CPU, 32GB of memory, and 5Gbps network bandwidth. Based on the data scale and processing complexity of the micro-batch, it is predicted that micro-batch A requires 4-core CPU, 16GB of memory, and 2Gbps of bandwidth; micro-batch B requires 2-core CPU, 8GB of memory, and 1Gbps of bandwidth; micro-batch C requires 6-core CPU, 24GB of memory, and 3Gbps of bandwidth. The ratio of the allocated resource amount to the total resource amount (resource utilization rate) collected for N1 is 50% for CPU, 60% for memory, and 40% for bandwidth, with an average of 50%; the resource utilization rate of N2 is 60% for CPU, 70% for memory, and 50% for bandwidth, with an average of 60%.

[0064] Divide the available resource vector of each processing node by the resource demand vector and multiply by the resource utilization rate to calculate the resource matching degree. For micro-batch A, the resource matching degree of N1 is (16 / 4)×(64 / 16)×(10 / 2)×50% = 10; the resource matching degree of N2 is (8 / 4)×(32 / 16)×(5 / 2)×60% = 6. The resource matching degrees of micro-batch B and C can be calculated.

[0065] Considering the load balancing factor, collect the CPU usage rate, memory usage rate, and network usage rate of the acquisition and processing nodes. Assume that the three usage rates of N1 are 40%, 45%, and 35% respectively, and the average value is 40%; the rates of N2 are 65%, 70%, and 60% respectively, and the average value is 65%. The average value of the loads of all processing nodes is (40% + 65%) / 2 = 52.5%, and the maximum system load is 100%. For N1, the load balancing factor is calculated as 1 - |(40% - 52.5%)| / 100% = 0.875; for N2, the load balancing factor is 1 - |(65% - 52.5%)| / 100% = 0.875.

[0066] Integrate multi-dimensional evaluation indicators and preset weight coefficients. Assume that the weights of the historical processing performance score, resource matching degree, and load balancing factor are 0.4, 0.4, and 0.2 respectively. Multiply the indicators by the weight coefficients and sum them up to obtain the affinity of the processing node. The affinity of micro-batch A for N1 is 0.4×114.38 + 0.4×10 + 0.2×0.875 = 50.15; the affinity for N2 is 0.4×95.13 + 0.4×6 + 0.2×0.875 = 41.23. The affinity of N1 is higher, so it is selected as the task migration target node for micro-batch A.

[0067] Multiply the execution time slice by the affinity of the processing node to obtain the comprehensive scheduling score. The comprehensive scheduling score of micro-batch A on N1 is 37.9×50.15 = 1900.69; the comprehensive scheduling score of micro-batch B on N2 is 13.6×43.56 = 592.42 (assuming the affinity of N2 for micro-batch B is 43.56); the comprehensive scheduling score of micro-batch C on N1 is 48.5×51.28 = 2487.08 (assuming the affinity of N1 for micro-batch C is 51.28).

[0068] Based on the comprehensive scheduling score, generate a scheduling sequence: micro-batch C → micro-batch A → micro-batch B, and distribute the corresponding micro-batch data to the selected processing nodes. Each micro-batch is processed within its allocated execution time slice, and the processing results are collected and returned to the request side. Continuously monitor the processing performance and update the historical database to provide a basis for subsequent scheduling optimization.

[0069] In this embodiment, the execution time slice is dynamically calculated based on the micro-batch data volume and processing priority, and weighted round-robin is used for initial allocation, achieving fair allocation of task resources and avoiding the resource starvation problem of high-priority tasks. The historical processing performance is weighted and evaluated through a time decay factor, making the impact of recent processing performance on the score greater, improving the timeliness of performance evaluation, and ensuring that the historical processing performance score can accurately reflect the actual processing ability of the processing node. By comprehensively considering multi-dimensional resource metrics such as CPU, memory, and network, an accurate resource profile is established. Through the comparison of the available resource vector and the required resource vector, the accurate calculation of resource matching degree is realized, avoiding performance loss caused by unreasonable resource allocation.

[0070] In an alternative embodiment, Saving the checkpoint snapshot and restoring the data synchronization state according to the checkpoint snapshot in case of a failure includes: Obtain the current timestamp as the checkpoint identifier, and organize the data shards, metadata information, and data processing status information being processed into a state snapshot. For each data shard, record the shard number, data content, processing progress, and timestamp information; for the metadata information, record the data source configuration, data table structure, index information, and statistical information; for the processing status, record the amount of processing completed, the current processing position, and the processing parameter configuration.

[0071] Before writing the state snapshot to persistent storage, compress and chunk the snapshot data to reduce storage overhead. Ensure data integrity by calculating the checksum of the state snapshot data. Write the compressed snapshot data in chunks to the disk according to the preset storage path, and at the same time record the checkpoint identifier, snapshot storage location, and check information in the checkpoint index table.

[0072] When it is detected that a processing node fails, obtain the latest valid checkpoint identifier from the checkpoint index table. Find the corresponding snapshot storage location according to the checkpoint identifier, read and decompress the snapshot data. Verify the integrity of the snapshot data to ensure that the data is not damaged. Restore the data shard information, metadata information, and processing status information in the snapshot to memory.

[0073] Re-initialize the data processing environment based on the restored state information, including reconstructing the data structure, restoring the processing progress, and parameter configuration. Resume data processing from the recorded processing position to ensure the continuity and consistency of data processing, update the data synchronization state, and ensure that subsequent data can be correctly synchronized.

[0074] Exemplarily, in a distributed log processing scenario, a checkpoint is created every 10 minutes. The checkpoint snapshot contains information such as the name of the log file being processed, the current line number of the log being processed, the parsed log entries, and the values of statistical counters. When a processing node fails, recovery is performed from the most recent checkpoint: the corresponding log file is reopened, the recorded line number position is located, the value of the statistical counter is restored, and log processing continues.

[0075] In a second aspect of the embodiments of the present invention, a Flink-based unstructured data synchronization system is provided, including: A first unit, configured to obtain the storage path of the unstructured data to be processed and perform sharding processing on the unstructured data to obtain sharded data blocks; A second unit, configured to calculate a feature score based on the access time interval and access times of the sharded data blocks and generate a feature score table, write the sharded data blocks with feature scores greater than a first threshold into the heap memory layer, write other sharded data blocks into the RocksDB layer, divide the sharded data blocks in the heap memory layer into multiple data partitions and generate an initial data partition table, collect data access latency and calculate a latency prediction value in combination with deep reinforcement learning, perform gradient iteration to generate a first optimization threshold and trigger data migration to obtain a data partition table; A third unit, configured to read the unstructured data based on the data partition table according to a preset buffer size and construct the read unstructured data into binary data of a metadata area and a data payload area; A fourth unit, configured to calculate the ratio of the data volume of the binary data to the processing time limit as a processing load factor and divide micro-batches, calculate the ratio of the data processing rate to the backlog data volume as a backpressure index value, when the backpressure index value exceeds a preset second threshold, perform a population iteration operation to obtain the parallelism parameter and resource quota corresponding to the maximum throughput and update them to a processing operator, allocate time slices according to weighted round-robin, generate a micro-batch scheduling sequence and perform data processing on the micro-batches; A fifth unit, configured to save a checkpoint snapshot and restore the data synchronization state according to the checkpoint snapshot in case of a failure.

[0076] In a third aspect of the embodiments of the present invention, an electronic device is provided, including: A processor and a memory for storing instructions executable by the processor, wherein the processor is configured to call the instructions stored in the memory to execute the method described above.

[0077] In a fourth aspect of the embodiments of the present invention, a computer-readable storage medium is provided, on which computer program instructions are stored, and when the computer program instructions are executed by a processor, the method described above is implemented.

[0078] The present invention may be a method, an apparatus, a system, and / or a computer program product. The computer program product may include a computer-readable storage medium having thereon computer-readable program instructions for performing various aspects of the present invention.

[0079] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements on some or all of the technical features; and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A method for unstructured data synchronization based on Flink, characterized in that, Including: Obtain the storage path of the unstructured data to be processed and perform sharding processing on the unstructured data to obtain sharded data blocks; Calculate the feature scores based on the access time intervals and access times of the sharded data blocks and generate a feature score table. Write the sharded data blocks with feature scores greater than the first threshold into the heap memory layer, and write the other sharded data blocks into the RocksDB layer. Divide the sharded data blocks in the heap memory layer into multiple data partitions and generate an initial data partition table. Collect the data access latency and calculate the latency prediction value in combination with deep reinforcement learning. Perform gradient iteration to generate the first optimization threshold and trigger data migration to obtain the data partition table; Based on the preset buffer size, read the unstructured data according to the data partition table and construct the unstructured data read into binary data of the metadata area and the data payload area; Calculate the ratio of the data volume of the binary data to the processing time limit as the processing load coefficient and divide it into micro-batches. Calculate the ratio of the data processing rate to the backlog data volume as the backpressure index value. When the backpressure index value exceeds the preset second threshold, perform a population iteration operation to obtain the parallelism parameter and resource quota corresponding to the maximum throughput and update them to the processing operator. Allocate time slices according to weighted round-robin, generate a micro-batch scheduling sequence and perform data processing on the micro-batches; Save the checkpoint snapshot and restore the data synchronization state according to the checkpoint snapshot in case of a failure.

2. The method according to claim 1, wherein Obtain the storage path of the unstructured data to be processed and perform sharding processing on the unstructured data to obtain sharded data blocks, including: Use the file system interface to obtain the storage path of the unstructured data, and the storage path includes the data storage location and the data file name; Shard the unstructured data according to the preset maximum shard size, sequentially traverse the unstructured data and divide the data blocks according to the byte size to generate multiple sharded data blocks; Calculate the data size and index position of each sharded data block and record the data size and index position in the shard index table.

3. The method according to claim 1, characterized in that, Calculate the feature scores based on the access time intervals and access times of the sharded data blocks and generate a feature score table. Write the sharded data blocks with feature scores greater than the first threshold into the heap memory layer, and write the other sharded data blocks into the RocksDB layer. Divide the sharded data blocks in the heap memory layer into multiple data partitions and generate an initial data partition table, including: Extract the access time intervals and access times of the sharded data blocks. Take the time difference between two adjacent accesses as the access time interval, and the number of requests for the sharded data block within the unit time as the access time. Add the product of the reciprocal of the access time interval and the preset first weight coefficient to the product of the access time and the preset second weight coefficient to obtain the feature score. The sum of the first weight coefficient and the second weight coefficient is 1. Combine the identifier, access time interval, access time, feature score, and current storage location identifier corresponding to the sharded data block to construct the feature score table; Calculate the average value of all feature scores in the feature score table, add the product of the average value and the smoothing factor to the product of the previously obtained threshold of the previous cycle and the complementary value of the smoothing factor to obtain the first threshold. Subtract the smoothing factor from 1 to obtain the complementary value of the smoothing factor. Write the sharded data blocks with feature scores greater than the first threshold into the heap memory layer, and write the sharded data blocks with feature scores less than or equal to the first threshold into the RocksDB layer; Group the sharded data blocks in the heap memory layer according to a fixed partition size, assign a unique identifier to each data partition, and write the partition number, start position offset, number of sharded data blocks, total partition data size, and current partition status of the data partition into the initial data partition table.

4. The method according to claim 1, characterized in that Collect data access latency and calculate the latency prediction value in combination with deep reinforcement learning. Execute gradient iteration to generate the first optimization threshold and trigger data migration to obtain the data partition table, including: Collect data access latency to obtain response time data. Extract the average latency, latency variance, and access frequency fluctuation degree in the response time data as the latency feature vector, and perform normalization processing on the latency feature vector to generate the latency feature matrix; Input the latency feature matrix, the current system load level, and the storage capacity utilization rate into the deep reinforcement learning network, extract the temporal features of data access to obtain long-term dependencies, and determine the migration rules of sharded data blocks between different storage layers based on the long-term dependencies; Use a sliding time window to segment and process the historical latency sequence, perform iterative training on the latency data within each time window to obtain the latency prediction value, execute gradient iteration to update the training parameters, calculate the dynamic adjustment factor based on the current system load level and storage capacity utilization rate, and add the product of the latency prediction value and the dynamic adjustment factor to the product of the first threshold of the previous iteration cycle and the complementary value of the dynamic adjustment factor to obtain the first optimization threshold; Calculate the geometric mean of the access latency change amount, access frequency, and storage overhead of the sharded data blocks as the migration priority score and generate a migration queue. Identify the sharded data blocks to be migrated from the migration queue according to the first optimization threshold, perform data migration on the sharded data blocks to be migrated, and update the initial data partition table based on the distribution information of the sharded data blocks after migration to obtain the data partition table.

5. The method according to claim 1, characterized in that Based on the preset buffer size, read unstructured data from the data partition table and construct the read unstructured data into binary data of the metadata area and the data payload area, including: Read unstructured data from the data partition table according to the preset buffer size, read the unstructured data in chunks and cache it in memory; Extract the file name, file size, file type, creation time, access permission, and storage location in the unstructured data to construct the metadata area, and construct the actual content information in the unstructured data into the data payload area; Add a delimiter identifier at the end of the metadata area, and splice the metadata area and the data payload area in the preset format sequence to generate binary data.

6. The method according to claim 1, wherein Calculate the ratio of the data volume of the binary data to the processing time limit as the processing load coefficient and divide it into micro-batches. Calculate the ratio of the data processing rate to the backlog data volume as the backpressure index value. When the backpressure index value exceeds a pre-set second threshold, perform a population iteration operation to obtain the parallelism parameter and resource quota corresponding to the maximum throughput and update them to the processing operator, including: Calculate the ratio of the data volume of the binary data to the processing time limit within a pre-set time window to obtain the processing load coefficient. Determine the number of micro-batches based on the system resource capacity, and divide the processing load coefficient by the number of micro-batches to divide the micro-batches; Calculate the ratio of the data processing rate to the backlog data volume to obtain the backpressure index value. When the backpressure index value exceeds a pre-set second threshold, perform a population iteration. Construct multiple population individuals containing parallelism parameters and resource quotas, encode the parallelism parameters and resource quotas of each population individual. The value range of the parallelism parameter is from the minimum core number of the system to the maximum core number of the system, and the value range of the resource quota is from the minimum resource limit of the system to the maximum resource limit of the system; Calculate the system throughput corresponding to each population individual as the individual fitness value. Select high-quality individuals from the population individuals through the roulette wheel algorithm, perform crossover operations and mutation operations on the parallelism parameters and resource quotas of the high-quality individuals to generate new population individuals, and retain the individual with the highest fitness in the population individuals as the optimal individual. Repeat the individual selection, crossover operation and mutation operation until the fitness value of the population individuals converges or reaches the maximum number of iterations; Obtain the parallelism parameter and resource quota corresponding to the global optimal individual from the population individuals and update them to the processing operator.

7. The method according to claim 1, characterized in that, Generate a micro-batch scheduling sequence according to weighted round-robin allocation of execution time slices and perform data processing on the micro-batches, including: Obtain the data volume of the micro-batch and the processing priority corresponding to each micro-batch, calculate the micro-batch weight, multiply the micro-batch weight by the pre-set scheduling period to obtain the execution time slice of the micro-batch, and perform an initial allocation of the execution time slice using weighted round-robin; Obtain the processing throughput of each processing node in the historical processing record, calculate the difference between the historical processing time and the current time to obtain the time decay factor, multiply the processing throughput by the time decay factor and accumulate them, and divide by the sum of the time decay factors to obtain the historical processing performance score corresponding to each processing node; Collect the CPU core number, memory capacity and network bandwidth of the processing node to obtain the available resource vector, predict the resource demand vector based on the data scale and processing complexity of the micro-batch, and collect the ratio of the allocated resource volume of the processing node to the total resource volume to obtain the resource utilization rate; Divide the available resource vector by the resource demand vector and multiply by the resource utilization rate to obtain the resource matching degree; collect the average value of the CPU usage rate, memory usage rate and network usage rate of the processing node to obtain the current load, subtract the average value of the loads of all processing nodes from the current load, divide by the maximum load of the system and take the absolute value, and subtract the calculation result from 1 to obtain the load balancing factor; Multiply the historical processing performance score, resource matching degree, and load balancing factor by the preset weight coefficients respectively and sum them to obtain the processing node affinity, and multiply the execution time slice by the processing node affinity to obtain the comprehensive scheduling score; Select the processing node with the maximum processing node affinity as the task migration target node, generate a micro-batch scheduling sequence based on the comprehensive scheduling score, and distribute the micro-batch data to the task migration target node, and perform data processing within the allocated execution time slice to obtain the processing result.

8. A Flink-based unstructured data synchronization system for implementing the method according to any one of the preceding claims 1-7, characterized in that, It includes: The first unit is used to obtain the storage path of the unstructured data to be processed and perform sharding processing on the unstructured data to obtain sharded data blocks; The second unit is used to calculate the feature score based on the access time interval and access times of the sharded data blocks and generate a feature score table, write the sharded data blocks with feature scores greater than the first threshold into the heap memory layer, and write other sharded data blocks into the RocksDB layer, divide the sharded data blocks in the heap memory layer into multiple data partitions and generate an initial data partition table, collect data access latency and calculate the latency prediction value in combination with deep reinforcement learning, perform gradient iteration to generate the first optimization threshold and trigger data migration to obtain the data partition table; The third unit is used to read the unstructured data based on the data partition table according to the preset buffer size and construct the read unstructured data into binary data of the metadata area and the data payload area; The fourth unit is used to calculate the ratio of the data volume of the binary data to the processing time limit as the processing load coefficient and divide it into micro-batches, calculate the ratio of the data processing rate to the backlog data volume as the backpressure index value, and when the backpressure index value exceeds the preset second threshold, perform a population iteration operation to obtain the parallelism parameter and resource quota corresponding to the maximum throughput and update them to the processing operator, allocate the execution time slice according to weighted round-robin, generate a micro-batch scheduling sequence and perform data processing on the micro-batches; The fifth unit is used to save the checkpoint snapshot and restore the data synchronization state according to the checkpoint snapshot in case of a failure.

9. An electronic device, characterized in that, It includes: A processor; A memory for storing instructions executable by the processor; Wherein, the processor is configured to call the instructions stored in the memory to execute the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having computer program instructions stored thereon, characterized in that, When the computer program instructions are executed by the processor, the method according to any one of claims 1 to 7 is implemented.

Images