Database distributed version synchronization and intelligent compression method

Abstract

The invention discloses a database distributed version synchronization and intelligent compression method and system, and relates to the technical field of distributed databases. The method comprises the following steps: a database fragment management mechanism in a distributed environment divides a database into a plurality of fragments, and each fragment independently maintains a versioned storage manager; a cross-node version coordination mechanism ensures the consistency of global version numbers; according to a multi-copy synchronization mechanism, the consistency of multiple copies is ensured by adopting a two-stage submission protocol; an intelligent copy selection algorithm; a distributed garbage recycling mechanism; the version compression mechanism is used for analyzing historical version data and storing the historical version data by adopting increment; intelligently combining strategies, and determining a compression strategy according to the data access frequency. According to the method, the problem of consistency of data version synchronization in a distributed environment is solved, and meanwhile, storage space occupation is reduced through intelligent compression.

Description

A Distributed Version Synchronization and Intelligent Compression Method for Databases Technical Field

[0001] This invention relates to the field of distributed database technology, specifically to a method and system for distributed version synchronization and intelligent compression of databases. Background Technology

[0002] With the rapid development of internet applications, distributed database systems have become the mainstream choice for processing large-scale data. However, data version synchronization in a distributed environment faces many challenges: data consistency between different nodes is difficult to guarantee; multi-replica management is highly complex; historical data occupies a large amount of storage space; and cross-node query performance is poor.

[0003] While existing distributed database systems such as Cassandra and MongoDB offer data replication and sharding capabilities, they fall short in version synchronization and compression. This is particularly true for applications requiring the preservation of complete historical versions, such as blockchain, supply chain traceability, and scientific experimental data management, where current technologies struggle to effectively manage historical data while maintaining performance.

[0004] While version compression technology can reduce storage space, it is difficult to implement in a distributed environment. Traditional compression algorithms often sacrifice query performance and struggle to guarantee data consistency after compression. Furthermore, effective solutions are lacking for issues such as distributed garbage collection and intelligent replica selection.

[0005] Therefore, a version synchronization and intelligent compression method specifically designed for distributed environments is needed to solve the above-mentioned technical problems.

[0006] This invention is a supplement to patent number 2022112701473. Summary of the Invention

[0007] This invention provides a database distributed version synchronization and intelligent compression method and system, which aims to solve the consistency problem of data version synchronization in a distributed environment, while reducing storage space usage through intelligent compression.

[0008] The technical solution of this invention includes: a database sharding management mechanism in a distributed environment; a multi-replica synchronization mechanism that uses a two-phase commit protocol to ensure consistency; a distributed garbage collection mechanism that coordinates all nodes to perform global garbage collection; a version compression mechanism that analyzes historical version data and uses incremental storage; and an intelligent merging strategy that determines the compression strategy based on data access frequency. Beneficial effects

[0009] Highly efficient data consistency guarantee: A two-phase commit protocol and multi-replica synchronization mechanism ensure data version consistency in a distributed environment. Cross-node version coordination ensures a unified global version number, avoiding version conflicts.

[0010] Intelligent storage space optimization: The version compression mechanism analyzes data change patterns and uses incremental storage for unchanged data, significantly reducing storage space usage. The intelligent merging strategy adjusts the compression granularity based on access frequency, balancing storage efficiency and query performance.

[0011] Optimized query performance: The intelligent replica selection algorithm selects the optimal replica based on node load and network conditions, reducing query latency. Geographically based path optimization further enhances the user experience.

[0012] Reliable disaster recovery: Regular backup and incremental backup mechanisms ensure data security, support data recovery based on any historical version, and minimize the risk of data loss.

[0013] Automated resource management: A distributed garbage collection mechanism coordinates resource reclamation across nodes to prevent memory leaks. Data lifecycle management automatically archives infrequently accessed data, reducing storage costs.

[0014] Flexible scalability: The sharding management mechanism supports horizontal scaling, allowing for dynamic addition of nodes based on data growth. The intelligent caching mechanism adaptively adjusts caching strategies to improve overall system performance. Detailed Implementation

[0015] Implementation Method 1: Distributed Version Coordinator Implementation (Rust) rust / / Distributed Version Coordinator use std::collections::{HashMap, HashSet}; use std::sync::{Arc, Mutex}; use tokio::sync::{RwLock, Semaphore}; pub struct DistributedVersionCoordinator {nodes: Arc <RwLock<HashMap<String, NodeInfo> >>,global_version: Arc <Mutex <u64>>,partitions: Arc<RwLock<HashMap<String, Partition>>>,raft_consensus: Option <raftconsensus>,semaphore: Arc <semaphore>,}impl DistributedVersionCoordinator { / / / Create a new coordinator pub async fn new(node_ids: Vec <string>) -> Self { let nodes = Self::initialize_nodes(node_ids).await; let partitions = Self::create_partitions(&nodes).await; Self { nodes: Arc::new(RwLock::new(nodes)), global_version: Arc::new(Mutex::new(0)), partitions: Arc::new(RwLock::new(partitions)), raft_consensus: Some(RaftConsensus::new().await), semaphore: Arc::new(Semaphore::new(100)), / / Concurrency control}} / / / Start a global transaction pub async fn begin_global_transaction(&self, changes: Vec <datachange>) -> Result<TransactionResult, CoordinatorError> { let _permit = self.semaphore.acquire().await?; let transaction_id = self.generate_transaction_id(); let target_version = { let mut version = self.global_version.lock().unwrap(); *version += 1; *version}; / / Phase 1: Prepare let prepared = self.prepare_all_partitions(&transaction_id, &changes, target_version).await?; if!prepared { self.rollback_all(&transaction_id).await?; return Err(CoordinatorError::PrepareFailed);} / / Phase 2: Commit self.commit_all_partitions(&transaction_id, target_version).await?; / / Record coordination log self.log_coordination(&transaction_id, "committed", target_version).await?; Ok(TransactionResult { success: true, version: target_version, transaction_id,})} / / / Prepare all shards async fn prepare_all_partitions(&self, transaction_id: &str, changes: &[DataChange], target_version: u64) -> Result<bool, CoordinatorError> { / / Group changes by shard let changes_by_shard = self.group_changes_by_shard(changes).await;let mut prepare_tasks = Vec::new();for (shard_id, shard_changes) in changes_by_shard {let partitions = self.partitions.read().await;if let Some(partition) = partitions.get(&shard_id) {let task = self.prepare_partition(partition,transaction_id,&shard_changes,target_version);prepare_tasks.push(task);}} / / Concurrent execution preparation let results = futures::future::join_all(prepare_tasks).await; / / Check all results let all_prepared = results.iter().all(|result| {matches!(result, Ok(true))});Ok(all_prepared)} / / / Prepare a single shard async fn prepare_partition(&self,partition: &Partition,transaction_id: &str,changes: &[DataChange],target_version: u64) -> Result<bool, CoordinatorError> { / / Send a prepare request to the primary node let primary_node = &partition.primary; let request = PrepareRequest {transaction_id: transaction_id.to_string(),changes: changes.to_vec(),target_version,timestamp: std::time::SystemTime::now().duration_since(std::time::UNIX_EPOCH).unwrap().as_millis() as u64,}; / / Use RPC to call let client = NodeClient::new(primary_node.address.clone()); match client.prepare(request).await {Ok(response) => Ok(response.prepared),Err(e) => {log::error!("Prepare failed for partition {}: {}",partition.shard_id, e);Ok(false)}}}} / / Raft consensus algorithm implementationstruct RaftConsensus {current_term: u64,voted_for: Option. <string>,log: Thing <logentry>, state: RaftState, election_timeout: std::time::Duration, heartbeat_interval: std::time::Duration,} impl RaftConsensus { async fn new() -> Self { Self { current_term: 0, voted_for: None, log: Vec::new(), state: RaftState::Follower, election_timeout: std::time::Duration::from_millis(150), heartbeat_interval: std::time::Duration::from_millis(50),}} / / / Handle the Request Vote RPC async fn handle_request_vote(&mut self, request: RequestVoteRequest) -> RequestVoteResponse { let mut granted = false; if request.term > self.current_term { self.current_term = request.term; self.state = RaftState::Follower; self.voted_for = None;} if request.term == self.current_term { if self.voted_for.is_none() || self.voted_for.as_deref() == Some(&request.candidate_id) { if self.is_log_up_to_date(&request) { granted = true; self.voted_for = Some(request.candidate_id.clone());}}} RequestVoteResponse { term: self.current_term, vote_granted: granted,}} / / / Check if the log is up-to-date fn is_log_up_to_date(&self, request: &RequestVoteRequest) -> bool { let last_log_entry = self.log.last(); if let Some(last_entry) = last_log_entry {request.last_log_term > last_entry.term ||(request.last_log_term == last_entry.term &&request.last_log_index >= self.log.len() as u64)} else {true}}} Implementation Method 2: Smart Replica Selector Implementation rust / / Smart Replica Selector pub struct SmartReplicaSelector {node_metrics: Arc <RwLock<HashMap<String, NodeMetrics> >>,selection_history: Arc <Mutex<Vec. <selectionrecord>>>,weights: SelectionWeights,geo_locator: Option <geolocator>, prediction_model: PredictionModel,} impl SmartReplicaSelector { pub fn new() -> Self { Self { node_metrics: Arc::new(RwLock::new(HashMap::new())), selection_history: Arc::new(Mutex::new(Vec::new())), weights: SelectionWeights::default(), geo_locator: GeoLocator::new().ok(), prediction_model: PredictionModel::new(),}} / / / Select the best replica pub async fn select_best_replica(&self, replicas: &[ReplicaInfo], operation: OperationType, client_location: Option <geolocation>) -> Result<ReplicaInfo, SelectionError> {if replicas.is_empty() {return Err(SelectionError::NoAvailableReplicas);} / / Calculate the score for each replica let mut scored_replicas = Vec::new();for replica in replicas {let score = self.calculate_score(replica, operation,client_location.clone()).await?;scored_replicas.push((replica.clone(), score));} / / Sort by score sorted_replicas.sort_by(|a, b| b.1.partial_cmp(&a.1).unwrap()); / / Select the best replica let (best_replica, best_score) = &scored_replicas[0]; / / Record the selection history self.record_selection(&best_replica, *best_score, operation).await; / / Regularly update the weights: self.update_weights().await; Ok(best_replica.clone())} / / / Calculate the replica score: async fn calculate_score(&self,replica: &ReplicaInfo,operation: OperationType,client_location: Option) <geolocation>) -> Result<f64, SelectionError> { let metrics = self.get_node_metrics(&replica.node_id).await; let mut score = 0.0; / / 1. Latency score let latency_score = self.calculate_latency_score(metrics.latency_ms); score += latency_score * self.weights.latency; / / 2. CPU load score let cpu_score = self.calculate_cpu_score(metrics.cpu_usage); score += cpu_score * self.weights.cpu_load; / / 3. Memory score let memory_score = self.calculate_memory_score(metrics.memory_usage); score += memory_score * self.weights.memory_load; / / 4. Network score let network_score = self.calculate_network_score(metrics.network_throughput); score += network_score * self.weights.network; / / 5. Reliability score let reliability_score = self.calculate_reliability_score(metrics.uptime, metrics.error_rate); score += reliability_score * self.weights.reliability; / / 6. Geographical location score if let (Some(geo_locator), Some(client_loc)) = (&self.geo_locator, &client_location) { if let Some(replica_loc) = &replica.location { let distance = geo_locator.calculate_distance(client_loc, replica_loc); let geo_score = self.calculate_geo_score(distance); score += geo_score * self.weights.geography;}} / / 7. Adjust operation type score = self.adjust_for_operation(score, operation); / / 8. Adjust prediction model let predicted_load = self.prediction_model.predict_load(&replica.node_id).await; score = self.adjust_for_predicted_load(score, predicted_load); Ok(score)} / / / Adjust score according to operation type fn adjust_for_operation(&self, score: f64, operation:OperationType) -> f64 {match operation {OperationType::Read => score * 1.2, / / Read operation prefers low latency OperationType::Write => score * 0.9, / / Write operation prefers high reliability OperationType::Scan => score * 1.1, / / Scan operations prefer high throughput OperationType::Transaction => score * 0.8, / / Transaction operations prefer strong consistency}} / / / Calculate latency score fn calculate_latency_score(&self, latency_ms: f64) -> f64 { / / Exponential decay function: the lower the latency, the higher the score if latency_ms < 1.0 {1.0} else if latency_ms < 10.0 {0.9} else if latency_ms < 50.0 {0.7} else if latency_ms < 100.0 {0.5} else if latency_ms < 200.0 {0.3} else {0.1}} / / / Record selection history async fn record_selection(&self, replica: &ReplicaInfo, score:f64, operation: OperationType) {let mut history = self.selection_history.lock().unwrap();history.push(SelectionRecord {timestamp: std::time::SystemTime::now(),replica_id: replica.id.clone(),node_id: replica.node_id.clone(),score,operation,success: None, / / will be updated after operation}); / / Keep the history size if history.len() > 10000 {history.drain(0..1000);}} / / / Update selection weights async fn update_weights(&self) {let history = self.selection_history.lock().unwrap();if history.len() < 1000 {return;} / / Analyze recent success rate let recent_history: Vec<_> = history.iter().rev().take(1000).collect(); let success_rates = self.analyze_success_rates(&recent_history); / / Adjust weights based on success rate let mut new_weights = self.weights.clone(); / / Increase latency weight if high-latency nodes have a high success rate if success_rates.high_latency_success > success_rates.low_latency_success * 1.1 {new_weights.latency *= 0.9;} / / Decrease CPU load weight if success_rates.high_cpu_success > success_rates.low_cpu_success * 1.1 {new_weights.cpu_load *= 0.9;}8;} / / Apply smooth update self.apply_weight_update(new_weights).await;}} Implementation method three: Intelligent version compressor to implement Rust / / Intelligent version compressor pub struct IntelligentVersionCompressor {compression_algorithms: HashMap <CompressionType, Box. <dyncompressionalgorithm>, analysis_engine: AnalysisEngine, access_tracker: AccessTracker, compression_policies: HashMap<DataCategory, CompressionPolicy>,} impl IntelligentVersionCompressor { pub fn new() -> Self { let mut compressor = Self { compression_algorithms: HashMap::new(), analysis_engine: AnalysisEngine::new(), access_tracker: AccessTracker::new(), compression_policies: HashMap::new(),}; / / Register compression algorithms compressor.register_algorithm(CompressionType::Delta, Box::new(DeltaCompression::new())); compressor.register_algorithm(CompressionType::Snapshot, Box::new(SnapshotCompression::new())); compressor.register_algorithm(CompressionType::Hybrid, Box::new(HybridCompression::new())); / / Initialize compression policies compressor.initialize_policies(); compressor} / / / Compress version history pub async fn compress_versions(&self, obj_id: &str, versions: Vec <versiondata>,target_compression_ratio: f64) -> Result<CompressedData, CompressionError> / / 1. Analyze version data let analysis = self.analysis_engine.analyze(&versions).await?; / / 2. Determine data category let data_category = self.classify_data(obj_id, &analysis).await; / / 3. Get compression policy let policy = self.get_compression_policy(&data_category, target_compression_ratio); / / 4. Select compression algorithm let algorithm = self.select_algorithm(&analysis, &policy); / / 5. Execute compression let compressed = algorithm.compress(&versions, &policy).await?; / / 6. Validate compression result self.validate_compression(&versions, &compressed, target_compression_ratio).await?; / / 7. Record compression information self.record_compression(obj_id, &analysis, &compressed, &policy).await; Ok(compressed)} / / / Analyze version data async fn analyze_versions(&self, versions: &[VersionData]) ->Result<VersionAnalysis, CompressionError> {let mut analysis = VersionAnalysis::new();if versions.len() < 2 {return Ok(analysis);} / / Calculate basic statistics analysis.total_versions = versions.len();analysis.total_size = versions.iter().map(|v| v.data.len()).sum(); / / Analyze change patterns let mut change_patterns = HashMap::new(); for i in 1..versions.len() { let prev = &versions[i - 1]; let curr = &versions[i]; / / Calculate differences let diff = self.calculate_diff(prev, curr); / / Analyze difference types let pattern = self.identify_change_pattern(&diff); *change_patterns.entry(pattern).or_insert(0) += 1; / / Update analysis data analysis.total_changes += diff.changed_size; analysis.change_frequency.push(diff.change_ratio);} analysis.change_patterns = change_patterns; analysis.compression_potential = 1.0 - (analysis.total_changesas f64 / analysis.total_size as f64); Ok(analysis)} / / / Determine the data category: `async fn classify_data(&self, obj_id: &str, analysis: &VersionAnalysis) -> DataCategory { / / Get access pattern let access_pattern = self.access_tracker.get_access_pattern(obj_id).await; / / Classify according to access frequency and change pattern if access_pattern.daily_access > 1000 {DataCategory::Hot} else if access_pattern.daily_access > 100 {if analysis.compression_potential > 0.7 {DataCategory::WarmCompressible} else {DataCategory::Warm}} else if access_pattern.daily_access > 10 {DataCategory::Cold} else {DataCategory::Archive} / / / Get compression policy fn get_compression_policy(&self,category: &DataCategory,target_ratio: f64) -> CompressionPolicy {let base_policy = self.compression_policies.get(category).cloned().unwrap_or_else(|| CompressionPolicy::default()); / / Adjust policy according to target compression ratio let mut policy = base_policy.clone(); if target_ratio < 0.5 { / / High compression requirement policy.merge_threshold = 0.7;policy.min_versions_to_merge = 10;policy.compression_level = 9;} else if target_ratio < 0.8 { / / Medium compression requirement policy.merge_threshold = 0.8;policy.min_versions_to_merge = 20;policy.compression_level = 6;} else { / / Low compression requirements (maintain performance)policy.merge_threshold = 0.9;policy.min_versions_to_merge = 50;policy.compression_level = 3;}policy} / / / Select compression algorithm fn select_algorithm(&self,analysis: &VersionAnalysis,policy: &CompressionPolicy) -> &Box<dyn CompressionAlgorithm> { / / Select the algorithm based on the analysis results if analysis.compression_potential > 0.9 { / / The change is small, use incremental compression self.compression_algorithms.get(&CompressionType::Delta).unwrap()} else if analysis.compression_potential > 0.9 { / / If the change is small, use incremental compression self.compression_algorithms.get(&CompressionType::Delta).unwrap()} else if analysis.compression_potential > 0.9compression_potential > 0.5 { / / Medium change, use hybrid compression self.compression_algorithms.get(&CompressionType::Hybrid).unwrap()} else { / / Large change, use snapshot compression self.compression_algorithms.get(&CompressionType::Snapshot).unwrap()}}} / / Incremental compression algorithm struct DeltaCompression {diff_engine: DiffEngine,compression_format: CompressionFormat,}impl DeltaCompression {fn new() -> Self {Self {diff_engine: DiffEngine::new(),compression_format: CompressionFormat::Zstd,}}}#[async_trait::async_trait]impl CompressionAlgorithm for DeltaCompression {async fn compress(&self,versions: &[VersionData],policy: &CompressionPolicy) -> Result<CompressedData, CompressionError> {if versions.is_empty() {return Ok(CompressedData::empty());}let mut compressed = CompressedData::new(); / / Select the base version (intermediate version as the base)let base_index = versions.len() / 2;let base_version = &versions[base_index];compressed.add_chunk(CompressedChunk {version: base_version.version,data: self.compress_data(&base_version).`data)?,chunk_type: ChunkType::Full,base_version: None,}); / / Store differences for other versions for (i, version) in versions.iter().enumerate() {if i == base_index {continue;} / / Calculate the difference from the base version let diff = self.diff_engine.compute_diff(&base_version.data, &version.data).await?;if diff.size < version.data.len() * policy.merge_threshold as usize { / / If the difference is small enough, store the difference compressed.add_chunk(CompressedChunk {version: version.version,data: self.compress_data(&diff.data)?,chunk_type: ChunkType::Delta,base_version: Some(base_version.version),});} else { / / If the difference is too large, store the complete data compressed.add_chunk(CompressedChunk {version: version.version,data: self.compress_data(&version.data)?,chunk_type: ChunkType::Full,base_version: None,});}} / / Build the compressed index compressed.build_index()?;Ok(compressed)}} Implementation Method 4: Distributed Garbage Collector Implementation rust / / Distributed Garbage Collector pub struct DistributedGarbageCollector {coordinators: Arc <RwLock<HashMap<String,GarbageCollectorCoordinator> >>, gc_state: Arc <Mutex. <garbagecollectionstate>>, communication_layer: CommunicationLayer, recovery_manager: RecoveryManager,}impl DistributedGarbageCollector { / / / Perform distributed garbage collection pub async fn perform_distributed_gc(&self) -> Result<GarbageCollectionResult, GarbageCollectionError> {log::info!("Starting distributed garbage collection");let start_time = std::time::Instant::now();let mut result = GarbageCollectionResult::new(); / / 1. Coordinate pause new reference creation self.coordinate_pause().await?; / / 2. Global reference analysis let global_analysis = self.analyze_global_references().await?;result.total_objects = global_analysis.total_objects;result.live_objects = global_analysis.live_count; / / 3. Identify recyclable objects let recyclable = self.identify_recyclable_objects(&global_analysis).await?;result.recyclable_objects = recyclable.len(); / / 4. Verify cross-node references self.verify_cross_node_references(&recyclable, &global_analysis.cross_refs).await?; / / 5. Create recycling batches (load balancing) let batches = self.create_balanced_batches(&recyclable, 100).await?; log::info!("Scheduling {} recycle batches", batches.len()); / / 6.Execute batch recycling for (i, batch) in batches.iter().enumerate() {log::info!("Processing batch {} / {}", i + 1, batches.len()); let batch_result = self.execute_recycle_batch(batch).await?; result.merge_batch_result(batch_result); / / Pause between batches to reduce system load if i < batches.len() - 1 {tokio::time::sleep(std::time::Duration::from_millis(50)).await;}} / / 7. Update state self.update_gc_state(GarbageCollectionState::Completed).await?; / / 8. Resume service self.coordinate_resume().await?; result.duration = start_time.elapsed(); result.success = true; log::info!("Distributed garbage collection completed: {:?}",result);Ok(result)} / / / Analyze global references async fn analyze_global_references(&self) -> Result<GlobalReferenceAnalysis, GarbageCollectionError> {log::info!("Analyzing global references");let mut analysis = GlobalReferenceAnalysis::new(); / / Collect reference information for all nodes let node_refs = self.collect_all_node_references().await?; / / Build the global reference graph let reference_graph = self.build_global_reference_graph(&node_refs).await?;analysis.total_objects = reference_graph.len(); / / Identify cross-node references analysis.}cross_refs = self.identify_cross_node_references(&reference_graph).await?; / / Calculate active objects analysis.live_count = self.compute_live_objects(&reference_graph).await?; log::info!("Global reference analysis completed: {} total objects, {} live objects", analysis.total_objects, analysis.live_count); Ok(analysis)} / / / Collect all node references async fn collect_all_node_references(&self) -> Result<Vec. <nodereferences>, GarbageCollectionError> { let coordinators = self.coordinators.read().await; let mut tasks = Vec::new(); for (node_id, coordinator) in coordinators.iter() { let task = self.collect_node_references(node_id, coordinator); tasks.push(task);} let results = futures::future::join_all(tasks).await; let mut all_refs = Vec::new(); for result in results { match result { Ok(refs) => all_refs.push(refs), Err(e) => { log::warn!("Failed to collect references from node: {}", e); / / 继续处理其他节点}}} Ok(all_refs)} / / / 识别跨节点引用 async fn identify_cross_node_references(&self, reference_graph: &HashMap<String, ObjectReferences>) -> Result<HashSet <string>, GarbageCollectionError> {let mut cross_refs = HashSet::new();for (obj_id, refs) in reference_graph {let mut involved_nodes = HashSet::new(); / / Collect the source node for source in &refs.referenced_by {involved_nodes.insert(source.node_id.clone());} / / Collect the node where the referenced object is located for target in &refs.references {involved_nodes.insert(target.node_id.clone());} / / If multiple nodes are involved, it is a cross-node reference if involved_nodes.len() > 1 {cross_refs.insert(obj_id.clone());}}log::info!("Identified {} cross-node references", cross_refs.len());Ok(cross_refs)} / / / Create a load-balanced recycling batch async fn create_balanced_batches(&self,recyclable: &[RecyclableObject],batch_size: usize) -> Result <Vec<Vec <recyclableobject>>, GarbageCollectionError> { / / Group by node let mut by_node: HashMap <String, Vec <recyclableobject>> = HashMap::new(); for obj in recyclable {by_node.entry(obj.node_id.clone()).or_insert_with(Vec::new).push(obj.clone());} / / Create load-balanced batches let mut batches = Vec::new(); let mut node_queues: HashMap <String, VecDeque <recyclableobject>> = HashMap::new(); / / Initialize queue for (node_id, objects) in by_node {node_queues.insert(node_id, objects.into());} / / Loop through each node to retrieve objects until all queues are empty while node_queues.values().any(|q| !q.is_empty()) { let mut batch = Vec::new(); for (node_id, queue) in node_queues.iter_mut() { if !queue.is_empty() { / / Retrieve some objects from this node (considering node weights) let take_count = self.calculate_batch_size_for_node(node_id, batch_size, &node_queues); let actual_take = take_count.min(queue.len()); let objects: Vec<_> = queue.drain(..actual_take).collect(); batch.extend(objects);} if batch.len() >= batch_size {break;}}if !batch.is_empty() {batches.push(batch);}}log::info!("Created {} balanced batches", batches.len());Ok(batches)} / / / Calculate the batch size of the node fn calculate_batch_size_for_node(&self,node_id: &str,total_batch_size: usize,node_queues: &HashMap <String, VecDeque <recyclableobject>>) -> usize { let active_nodes = node_queues.values().filter(|q|!q.is_empty()).count(); if active_nodes == 0 { return 0;} / / Basic allocation let base_allocation = total_batch_size / active_nodes; / / Adjust according to node performance let performance_factor = self.get_node_performance_factor(node_id); (base_allocation as f64 * performance_factor).ceil() as usize}}< / recyclableobject> < / recyclableobject> < / recyclableobject> < / recyclableobject> < / string> < / nodereferences> < / garbagecollectionstate> < / versiondata> < / dyncompressionalgorithm> < / geolocation> < / geolocation> < / geolocator> < / selectionrecord> < / logentry> < / string> < / datachange> < / string> < / semaphore> < / raftconsensus>

Claims

1. A database distributed version synchronization and intelligent compression method, characterized in that, Includes the following steps: S1. Database Sharding Management in a Distributed Environment: S1.1 Divide the database into multiple shards, with each shard independently maintaining a versioned storage manager; S1.2 Establish a cross-node version coordination mechanism to ensure global version number consistency; S1.3 Implements a data synchronization protocol between shards to maintain data consistency; S2. Multi-replica synchronization mechanism: S2.1 Adopts a multi-replica strategy, with each data block having a replica stored on multiple nodes; S2.2 When a version changes, a two-phase commit protocol is used to ensure multi-replica consistency; S2.3 Implement an intelligent replica selection algorithm to select the optimal replica to provide services based on node load; S3. Distributed garbage collection mechanism: S3.1 Coordinate global reference counting analysis among nodes; S3.2 Identify data blocks referenced across nodes to prevent erroneous garbage collection; S3.3 Distributed garbage collection task scheduling to avoid impacting system performance; S4. Version compression mechanism: S4.1 Periodically analyze historical version data to identify data change patterns; S4.2 For data that has not changed in multiple consecutive versions, incremental storage is used; S4.3 merges and stores similar data from adjacent versions, reducing storage space usage.

2. The method according to claim 1, characterized in that, The multi-replica synchronization mechanism also includes data consistency verification: S2.4 Periodically check the data consistency between replicas; S2.5 When inconsistency is detected, automatically trigger the data repair process; S2.6 Record the replica synchronization history to support fault diagnosis and performance analysis.

3. The method according to claim 1, characterized in that, The version compression mechanism also includes an intelligent merging strategy: S4.4 Determine the compression strategy for different data based on the data access frequency: a) High-frequency access data: maintain finer-grained versions; b) Low-frequency access data: merge versions with similar access times; S4.5 Establish a compressed version index to ensure that any historical version can still be quickly accessed after compression.

4. The method according to claim 1, characterized in that, It also includes transaction processing mechanisms: S5.1 supports multi-version concurrency control (MVCC), so read operations do not block write operations; S5.2 implements atomic commit, ensuring that all operations within a transaction take effect simultaneously; S5.3 provides isolation level configuration to support different consistency requirements.

5. The method according to claim 1, characterized in that, It also includes disaster recovery mechanisms: S6.1 regularly backs up the version index and critical data to persistent storage; S6.2 implements incremental backups, backing up only changes since the last backup; S6.3 supports data recovery based on any historical version, minimizing data loss.

6. The method according to claim 1, characterized in that, It also includes data lifecycle management: S7.1 Set different version retention policies based on data importance; S7.2 Automatically archive infrequently accessed historical versions to low-cost storage; S7.3 Provide data compliance auditing functions to ensure compliance with regulatory requirements.

7. The method according to claim 1, characterized in that, It also includes an intelligent caching mechanism: S8.1 Analyzes data access patterns and identifies popular historical versions; S8.2 Caches popular version data to high-speed storage media; S8.3 Implements cache preheating and intelligent eviction strategies.

8. The method according to claim 1, characterized in that, The cross-node version coordination mechanism adopts a distributed consensus algorithm, including: S1.4 Electing a master node using the Raft or Paxos algorithm; S1.5 The master node is responsible for allocating a global version number; S1.6 The slave nodes synchronize the version status of the master node.

9. The method according to claim 1, characterized in that, The intelligent replica selection algorithm includes: S2.7 Real-time monitoring of the performance indicators of each node, including CPU utilization, memory utilization, and network latency; S2.8 Dynamically adjusting the replica selection weights based on the performance indicators; S2.9 Implementing geographical location-based routing optimization to select the nearest replica to serve user requests.

10. A database system, characterized in that, include: Multiple server nodes, each node being configured with a server as described in claim 10 of the first patent; The network communication module is used for data synchronization and coordination between nodes; A load balancer is used to distribute user requests according to a smart replica selection algorithm; A backup storage system for storing backup data and historical versions; the system implements the method as described in any one of claims 1-9.