A cross-center storage-computing joint scheduling system and method based on data proximity

Abstract

The application discloses a cross-center storage-computing joint scheduling system and method based on data proximity, comprising the following steps: declaring the data set, model and code dependency of a task through unified annotation and generating a dependency graph; performing cross-center data source optimization discovery based on the dependency graph, and performing pre-checking before migration to evaluate feasibility, cost and risk; performing safe and reliable data migration by using an encryption signature link and a multi-layer verification mechanism; dynamically selecting the optimal computing center and node for task scheduling based on a joint cost function that combines computing cost, data migration cost, cross-center communication cost and data reuse benefit; and continuously optimizing the scheduling weight parameter according to historical data. The system comprises functional modules corresponding to the above steps. The application realizes global joint optimization of storage and computing resources, effectively reduces task cold start time and cross-center transmission cost, improves resource utilization efficiency, and guarantees data consistency and access security.

Description

Technical Field

[0001] This invention relates to the fields of cloud computing and distributed computing technology, specifically to a cross-center storage and computing joint scheduling system and method based on data proximity. Background Technology

[0002] With the rapid development of cloud computing, big data, and artificial intelligence technologies, distributed computing tasks increasingly rely on cross-datacenter resource allocation and collaborative execution. Against this backdrop, efficiently scheduling computing tasks and managing their dependent data objects has become a key challenge for improving overall system performance, reducing costs, and ensuring service quality.

[0003] Currently, the industry widely uses container orchestration platforms, such as Kubernetes, for application deployment and resource scheduling. The default Kubernetes scheduler primarily allocates tasks based on the availability of computing resources such as CPU, memory, and GPU on nodes, and supports topology constraints based on node affinity and anti-affinity. To further meet the needs of high-performance computing and AI training scenarios, scheduling frameworks such as Volcano and Kubeflow have emerged, extending advanced scheduling strategies such as queue management, resource quotas, and task preemption. However, these scheduling systems mainly focus on matching and balancing computing resources during the decision-making process, failing to incorporate data location, migration costs, and consistency requirements into a unified optimization objective. This leads to a prominent cold start delay phenomenon of "computation waiting for data," especially in large-scale cross-center scenarios, where task startup time increases significantly.

[0004] In terms of data management, current systems typically employ a separate declaration and access approach. Datasets may be mounted via persistent volume declarations (PVCs) or object storage services (such as Amazon S3 and Alibaba Cloud OSS); model files are usually packaged in container images or distributed via object storage; and code is injected into the container during initialization via a Git repository. This decentralized data declaration mechanism lacks a unified semantic description framework, making it difficult for the scheduler to globally perceive the data dependency graph of tasks, and consequently, to comprehensively consider data readiness during scheduling. Furthermore, cross-datacenter data synchronization often relies on the asynchronous replication mechanism of the storage system itself (such as cross-cluster replication of Ceph and HDFS). Such mechanisms typically provide eventual consistency or weak consistency guarantees, failing to ensure data integrity and version correctness before task startup. Data errors are often only exposed during task execution, leading to task failure or unreliable results.

[0005] At the storage scheduling level, existing solutions often treat "data migration" and "compute scheduling" as two separate subsystems. Data migration strategies are mostly statically configured or triggered by simple rules, lacking dynamic assessment of network conditions, storage load, and cost models. Furthermore, the lack of end-to-end data integrity verification mechanisms means that silent corruption may occur during transmission, and the migration task itself lacks idempotency and recoverability design, easily leading to duplicate transmissions or inconsistent states after failures, further wasting bandwidth and storage resources.

[0006] Furthermore, the existing scheduling system lacks systematic modeling and optimization of "data proximity." "Data proximity" refers to fully considering the network overhead between the current location of data and the computing node during scheduling decisions, thereby making a globally optimal choice among various strategies such as "migrating data to the computing node," "scheduling computing tasks to the node where the data resides," and "a compromise between partial migration and remote access." Because it fails to incorporate data migration costs, cross-center communication latency, and bandwidth costs into a unified cost model, the existing system struggles to achieve an effective balance between resource utilization, task execution time, and operational economics. Summary of the Invention

[0007] To address the problems of fragmented storage and computing decisions, insufficient consideration of data proximity, and weak consistency and security guarantees in existing technologies when scheduling across data centers, this invention provides a cross-center storage and computing joint scheduling system and method based on data proximity.

[0008] To achieve the above objectives, the present invention provides the following technical solution:

[0009] A cross-center storage and computing joint scheduling system based on data proximity includes:

[0010] The unified data declaration module is used to receive and parse the data dependency information declared in Kubernetes Pod annotations. The annotations encapsulate the attributes and scheduling strategies of three types of data objects required by the task—dataset, model, and code—in a unified format, and generate a structured data dependency graph for scheduling decisions.

[0011] The storage orchestration and verification module, connected to the unified data declaration module, is used to perform cross-data center data object discovery based on the data dependency graph through the unified remote file system (URFS), select the best option based on multi-dimensional scores of network quality, source load, reliability and cost, and implement multi-layer data consistency verification including at least transport layer block verification and object layer end-to-end verification.

[0012] The two-stage migration decision module, connected to the storage orchestration and verification module, is used to perform a pre-check assessment before data migration. The assessment includes at least migration feasibility, estimated transmission time and overall cost. During the execution phase, the hash value generated by the object version, task ID, target center and time window is used as the idempotent key to ensure the idempotency and recoverability of the migration operation.

[0013] The secure access verification module, connected to the storage orchestration and verification module, is used to generate a signature link based on cryptographic signatures with access scope and timeliness restrictions, and to implement multi-layered secure access and integrity verification of data objects based on this link, including HTTP header verification, range request verification and end-to-end verification.

[0014] The in-memory computing joint scheduling module is connected to the unified data declaration module and the two-stage migration decision module, respectively. It is used to optimize the solution based on the data dependency graph, the resource status of each data center and the network status, and a joint cost function that integrates computing cost, data migration cost, cross-center communication cost and data reuse benefits. It dynamically selects the optimal computing center and computing node, and continuously learns and adjusts the weight parameters in the cost function based on historical execution data.

[0015] Preferably, in the unified data declaration module, the data dependency information includes at least one of the following data objects: dataset, model, and code; each data object declares at least one of its location identifier, version information, size information, and verification summary through annotations;

[0016] The annotation also includes strategy parameters, which include at least one of data proximity requirements, storage strategy, and consistency level.

[0017] The unified data declaration module also includes an adapter component, which is used to parse the annotation and generate a structured data dependency graph. The data dependency graph is used to identify the dependencies between data objects, calculate the data acquisition priority, and evaluate the data access pattern.

[0018] Preferably, the unified remote file system includes:

[0019] The multi-source scanning unit is used to query the object storage of each data center in parallel, collect object metadata, and build a global location index.

[0020] The availability assessment unit is used to detect the network connectivity, historical access success rate, and response time of each storage source.

[0021] The optimal decision-making unit is used to select the optimal source from multiple available data sources based on a scoring function.

[0022] The verification execution unit is used to implement at least two of the following: transport layer block verification, object layer end-to-end verification, and service layer semantic verification.

[0023] Preferably, the source selection scoring function used by the optimal decision-making unit is:

[0024] ;

[0025] in, Calculated based on round-trip time and available bandwidth; Calculated based on the current number of transmission tasks; Calculated based on historical access success rate; Calculated based on storage and transmission costs; , , , These are configurable weighting coefficients.

[0026] Preferably, the two-stage migration decision module includes:

[0027] The pre-inspection unit is used to assess the data migration transfer time, overall cost, and risk score before performing the migration;

[0028] The execution unit is used to perform the actual migration after the pre-check passes, and adopts an idempotency guarantee mechanism, including a hash value generated based on the object version, task ID, target center and time window as an idempotency key, which is used to uniquely identify the migration task and prevent duplicate execution.

[0029] The execution unit also includes a state checkpoint mechanism and a conflict detection mechanism; the state checkpoint mechanism is used to record the state of key steps in the migration process and supports recovery from breakpoints; the conflict detection mechanism is used to detect and handle resource conflicts between concurrent migration tasks.

[0030] Preferably, the signature link generated by the secure access verification module is an encrypted string based on the HMAC-SHA256 algorithm, and contains the following information: signature value, expiration time, issuer, access scope, and allowed HTTP methods;

[0031] The multi-layered security access and integrity verification performed by the secure access verification module includes:

[0032] First-level verification: Obtain and verify object metadata based on HTTP HEAD requests;

[0033] Second-level verification: Perform block-based verification on the object based on the HTTP Range request;

[0034] Third-level verification: Download the complete object and perform end-to-end digest verification.

[0035] Preferably, the joint cost function used by the in-memory computing joint scheduling module is:

[0036] ;

[0037] in, To calculate resource costs, including queue waiting time, resource fragmentation penalty, and node load factor; The cost of migrating data to the target center includes transmission time costs, verification overhead, and storage rental costs; Additional costs associated with cross-center communication include distance penalties, latency penalties, and compliance costs; The revenue generated from data already existing or reusable in the target center includes cache hit revenue, sharing revenue, and locality reward; α, β, γ, and δ are dynamically adjustable weighting coefficients.

[0038] The in-memory computing joint scheduling module also includes a weight learning unit, which is used to collect historical scheduling decisions and execution results, and dynamically update the weight coefficients α, β, γ, and δ using the gradient descent method.

[0039] This invention also discloses a cross-center storage and computing joint scheduling method based on data proximity, comprising the following steps:

[0040] S1: Unified Declaration and Resolution of Data Dependencies: Receives Kubernetes Pod annotations carrying data dependency declarations. These declarations encapsulate the location, version, verification information, and scheduling strategy of three types of data objects required by the task: dataset, model, and code, in a unified format. Parses the annotations and generates a structured data dependency graph for the scheduler to globally evaluate data dependencies.

[0041] S2: Data source selection and pre-check: Based on the data dependency graph, multiple data centers are scanned in parallel through a unified remote file system to discover available replicas of each data object; a data source is selected from multiple replicas based on a comprehensive scoring function, which considers at least network quality, source load, and transmission cost; before triggering the actual data migration, a pre-check is performed to assess the feasibility of the migration, estimate the transmission time, and the overall cost.

[0042] S3: Secure Migration and Integrity Verification: If the pre-check passes, an encrypted signature link with access scope and time limits is generated; data migration is performed based on this link, and multi-layer integrity verification including chunked transmission verification and end-to-end digest verification is implemented during the migration process;

[0043] S4: Joint scheduling and task execution of storage and computing: Based on the data dependency graph, pre-check evaluation results, real-time resource status and network conditions of each data center, construct and solve the joint cost function, and use the joint cost function to calculate the total cost of each candidate scheduling; select the computing center and computing node with the optimal total cost according to the solution results; finally, schedule the computing task to the selected node for execution.

[0044] Preferably, the optimal selection and pre-check in step S2 specifically include:

[0045] S201: Multi-Source Discovery and Scoring: Parallel queries of object storage in each data center to obtain metadata, version, and location information of data objects; for each available data source, its source selection score is calculated using the following formula:

[0046] ;

[0047] in, Calculated based on round-trip time and available bandwidth; Calculated based on the current number of transmission tasks; Calculated based on historical access success rate; Calculated based on storage and transmission costs; , , , These are configurable weighting coefficients;

[0048] S202: Three-layer verification mechanism: Three-layer consistency verification is performed during data transmission, including: real-time digest verification of each data block at the transport layer; end-to-end verification of the complete object using MD5 or SHA256 at the object layer; and semantic compliance verification based on data format and dependencies at the business layer.

[0049] S203: Two-stage decision-making: In the pre-check phase, calculate:

[0050] ;

[0051] The migration risk score is assessed; the execution phase is initiated only when the estimated time and risk are below the threshold. During the execution phase, a hash value generated from the object version, task ID, target center, and time window is used as an idempotent key to ensure the uniqueness and recoverability of the migration operation.

[0052] Preferably, the in-memory computing joint scheduling decision in step S4 specifically includes:

[0053] S401: Define the joint cost function as follows:

[0054] ;

[0055] in, To calculate resource costs, including queue waiting time, resource fragmentation penalty, and node load factor; The cost of migrating data to the target center includes transmission time costs, verification overhead, and storage rental costs; Additional costs associated with cross-center communication include distance penalties, latency penalties, and compliance costs; The revenue generated from data already existing or reusable in the target center includes cache hit revenue, sharing revenue, and locality reward; α, β, γ, and δ are dynamically adjustable weighting coefficients.

[0056] S402: Dynamic Parameter Learning: Collect historical scheduling decisions and their corresponding actual task execution performance data, calculate the deviation between predicted costs and actual costs; use gradient descent to dynamically update and optimize the weight parameters α, β, γ, δ so that the scheduling decisions continuously adapt to changes in cluster load and network environment.

[0057] This invention provides a cross-center storage and computing joint scheduling system and method based on data proximity, which has the following beneficial effects:

[0058] By introducing a unified data declaration mechanism based on Kubernetes Pod annotations, the descriptions and strategies for three types of data are standardized and encapsulated. This enables the system to automatically parse and generate a structured data dependency graph, providing the scheduler with a global and accurate view of data dependencies. Consequently, operations personnel only need to master a simple annotation syntax to declare all data objects, eliminating the need for complex and error-prone configurations across multiple systems.

[0059] By employing a two-stage data migration decision-making mechanism, the system simulates and evaluates the feasibility, time, cost, and risks of the migration plan in the pre-check phase, and feeds the results back to the scheduling decision, avoiding scheduling tasks to centers where data migration is infeasible or extremely costly. In the execution phase, mechanisms such as idempotent keys, state checkpoints, and conflict merging ensure the reliability and efficiency of the migration operation.

[0060] The in-memory computing joint scheduling module employs a joint cost function that integrates computational cost, data migration cost, cross-center communication cost, and data reuse benefits. Before scheduling decisions, a migration pre-check is performed, and a heuristic algorithm is used to find the global optimum. The learning mechanism for weight parameters enables the system to dynamically adapt to environmental changes. Through pre-checking and intelligent scheduling, tasks are more likely to start on nodes where data is ready or where migration costs are lowest, significantly shortening the time from task submission to execution. The joint cost function comprehensively considers computational resource load, data migration cost, and cross-center communication overhead, dynamically guiding tasks to the resource combination with the lowest overall cost and highest efficiency. This reduces idle computational resources and data transmission bandwidth contention, improving the overall throughput and resource utilization of the cross-center cluster. By reducing unnecessary or inefficient cross-center data migration, network bandwidth usage costs are directly reduced.

[0061] By implementing a three-layer verification system in the storage orchestration and verification module—transport layer block verification, object layer end-to-end digest verification, and business layer semantic verification—combined with the proactive verification process of the secure access verification module, data corruption, version errors, or format inconsistencies can be promptly detected and corrected during data acquisition and migration. This prevents defective data from entering the computation stage, significantly reducing runtime task failures or incorrect results caused by data issues. The layered verification mechanism can quickly locate consistency problems at each stage of the data flow, shifting problem detection from reactive troubleshooting after task failure to the data preparation stage, greatly shortening the mean time to recovery. Furthermore, every data migration, verification, and use is recorded completely and verifiably, forming a clear data lineage and enhancing the traceability and reliability of computation results.

[0062] The secure access verification module generates encrypted signature links with strict temporal, spatial, and operational scope restrictions for use by third-party verification agents. These agents can perform multi-layered data verification without needing to know the permanent key. This reduces the risk of core access credentials being exposed during transmission and use, aligning with best security practices. Simultaneously, a standardized and auditable external data verification process is established, with detailed logging of all accesses via the signature links, providing a complete chain of evidence to meet data security and compliance audit requirements. Attached Figure Description

[0063] To more clearly illustrate the technical solutions in this invention or the prior art, the accompanying drawings used in the description of this invention or the prior art will be briefly introduced below.

[0064] Figure 1 This is a diagram of the overall system architecture of the present invention;

[0065] Figure 2This is a schematic diagram illustrating the working principle of the unified data declaration module in this invention.

[0066] Figure 3 This is a flowchart of the storage, orchestration, and verification module in this invention;

[0067] Figure 4 This is a flowchart of the two-stage migration decision module in this invention;

[0068] Figure 5 This is a flowchart of the in-memory computing joint scheduling decision-making process in this invention;

[0069] Figure 6 This is a flowchart illustrating the steps of the cross-center storage and computing joint scheduling method of the present invention. Detailed Implementation

[0070] To make the objectives, technical solutions, and advantages of the present invention clearer, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings.

[0071] Example 1, as Figures 1 to 6 As shown in the figure, the cross-center storage and computing joint scheduling system based on data proximity provided by the present invention is divided into four layers: user interface layer, scheduling decision layer, storage orchestration layer and infrastructure layer.

[0072] The user interface layer is responsible for receiving computational tasks (such as AI training jobs or online inference services) submitted by users using standard Kubernetes Pod templates. Users specify the datasets, models, and code required for the task by adding data dependency declarations in a standardized format to the metadata.annotations field of the Pod.

[0073] The scheduling decision layer, acting as the "brain" of the system, is the core of this invention. It comprises a unified data declaration module and a storage-compute joint scheduling module. The former parses user declarations, while the latter integrates global information and performs optimized scheduling based on data proximity awareness.

[0074] The storage orchestration layer is responsible for ensuring the flow and consistency of data across data centers. It integrates storage orchestration and verification modules, a two-stage migration decision module, and a secure access verification module, and specifically performs a series of operations such as data discovery, optimization, migration, and verification.

[0075] The infrastructure layer consists of Kubernetes clusters distributed across multiple geographical regions (data centers A, B, and C) and their corresponding object storage (such as S3 and OSS), providing computing, storage, and network resources for the upper layers.

[0076] The layers communicate with each other through well-defined interfaces. When a Pod is submitted, the scheduling decision layer triggers a cross-center storage-compute joint scheduling process.

[0077] in:

[0078] The unified data declaration module is used to receive and parse the data dependency information declared in Kubernetes Pod annotations. The annotations encapsulate the attributes and scheduling strategies of the three types of data objects required by the task—dataset, model, and code—in a unified format, and generate a structured data dependency graph for scheduling decisions.

[0079] Specifically, data dependencies are declared by adding a key-value pair to the `metadata.annotations` field of the user's Pod definition file when submitting a computation task. This key is typically named `data.dependencies`, and its value is a structured string conforming to YAML or JSON format. In this declaration, the user must categorize all data objects required by the task into three main categories: datasets, models, and code, according to predefined specifications, and fill in specific entries for each category. Each data object entry must contain a series of attributes, among which location identifier and version information are required. The location identifier describes the source address of the data and must use a recognizable Uniform Resource Locator (URL) format, supporting various protocols such as `s3: / / ` and `oss: / / ` for object storage paths, `git: / / ` for Git repository addresses, and `http: / / ` or `https: / / ` for network-accessible resources. Version information is used to precisely specify the specific version of the required data, supporting various forms such as semantic version numbers (e.g., v1.2.0), Git commit hashes (e.g., commit-abc123), or timestamps.

[0080] In addition to the required information, each data object entry may optionally include attributes such as size information and a checksum. The checksum is a key field used to verify data integrity, typically filled with a digest value calculated using hash algorithms such as MD5 or SHA-256. To guide the scheduling system in making optimization decisions, each data object entry must also include a set of policy parameters. These parameters include: data proximity requirements (e.g., "must be localized," "prefer localized," or "remotely accessible"), storage policies (e.g., "permanently reside after migration," "caching only," or "create replicas"), and consistency levels (e.g., "strict consistency," "eventual consistency"). These policies provide explicit semantic constraints and objectives for subsequent scheduling and migration decisions.

[0081] Additionally, a dedicated adapter component within the system (Annotation Adapter) is responsible for listening to and parsing Pod annotations. When a Pod containing the `data.dependencies` annotation is detected as being created, the adapter first extracts and parses the structured string, converting it into a series of internal data object models in memory. Subsequently, based on these data object models and their declared dependencies (e.g., a model file might depend on a specific version of a dataset), the adapter constructs a structured data dependency graph. This graph is typically represented as a directed acyclic graph (DAG), where nodes represent independent data objects, and edges represent dependencies or references between objects. By analyzing this graph, the system can clearly identify the order in which data is acquired and the possibility of parallel downloads.

[0082] Furthermore, the adapter extracts the corresponding policy parameters from each data object model and associates them with the data dependency graph. Finally, the adapter encapsulates the generated structured information containing complete dependencies and policy semantics into a standardized internal representation and passes it to the scheduling decision layer. This process transforms the originally scattered, implicit data dependencies into explicit information that the system can globally understand and process uniformly, laying a solid foundation for subsequent in-memory compute joint optimization scheduling.

[0083] The storage orchestration and verification module, connected to the unified data declaration module, is used to perform cross-datacenter data object discovery based on the data dependency graph and through the Unified Remote File System (URFS). It selects data objects based on a multi-dimensional score considering network quality, source load, reliability, and cost, and implements multi-layered data consistency verification, including at least transport layer block verification and object layer end-to-end verification. Specifically, the core of the storage orchestration and verification module is the Unified Remote File System service component deployed in each data center. Internally, this module completes the entire process from data discovery to successful delivery through the collaborative work of a multi-source scanning unit, an availability assessment unit, an optimization decision unit, and a verification execution unit. When the scheduling system determines that a specific data object needs to be acquired, this module initiates the collaborative workflow:

[0084] The multi-source scanning unit initiates the process, launching lightweight metadata queries in parallel to the storage systems of all participating data centers. The corresponding units in each center retrieve local storage and return metadata information such as the existence, size, and version of the target object to the requester. After collecting all possible replica location information, the availability assessment unit performs a multi-dimensional operational status assessment of each available data source. This unit dynamically measures the network round-trip latency and available bandwidth of each source to assess network quality, counts the number of data transmission tasks currently being processed by each source to determine its real-time load, queries historical request records to calculate the access success rate, and estimates transmission and storage costs based on the cloud service provider's billing model. The optimal decision-making unit receives the output from the availability assessment unit and applies a comprehensive scoring function for quantitative decision-making. This function normalizes the assessment scores of the four dimensions and performs a weighted sum according to preset, configurable weights (e.g., network quality has the highest weight, followed by cost), calculating a comprehensive score for each source. Finally, this unit selects the data source with the highest comprehensive score as the target for this data acquisition. This mechanism intelligently achieves load balancing and optimal performance selection of global data sources.

[0085] To ensure data integrity after transmission and storage across networks, the verification execution unit implements a rigorous three-layer verification system:

[0086] Real-time block checksum verification at the transport layer: During data transmission, the sender divides the data stream into fixed-size blocks and calculates a checksum for each block. Upon receiving each block, the receiver immediately calculates its checksum using the same algorithm and compares it. If a mismatch is found, a retransmission of that specific block is immediately requested. This mechanism can detect and correct bit errors caused by network noise early in the transmission process, avoiding the situation where an overall error is reported only after invalid data transmission has been completed, thus greatly improving the reliability of the transmission process.

[0087] Object-level integrity digest verification: After the entire object is transmitted, the receiver calculates the cryptographic hash digest of the complete object. This calculated digest value is then compared to the trusted original digest value provided by the user or upstream system in the data claim. Only when the two are completely identical is it confirmed that the object has not been tampered with or corrupted during transmission, signifying that object-level integrity has been verified.

[0088] Business-layer semantic compliance verification: For data with specific structures and business meanings, the system provides an extensible plugin mechanism for deeper verification. For example, for a machine learning model file, the verification plugin can check whether its format conforms to standards, whether the model architecture is complete, and whether the definitions of input and output tensors match expectations. This layer of verification goes beyond bit-level integrity, ensuring the availability and correctness of data in business logic.

[0089] For example, when a request to retrieve the object oss: / / model-repo / resnet50.tar.gz is received, the Unified Remote File System client will send metadata queries to the Unified Remote File System servers in all data centers in parallel. Each server checks its local storage and returns the object's metadata (existence / non-existence, size, last modified time, etc.). After collecting all responses, the client proceeds to the optimal decision-making process.

[0090] Scoring calculation: For each source that owns a copy of the object, according to the formula: Perform the calculations. Among them, Calculated based on round-trip time and available bandwidth; Calculated based on the current number of transmission tasks; Calculated based on historical access success rate; Based on storage and transmission cost calculations; , , , These are configurable weighting coefficients. For example, network quality. The load score is obtained by normalizing the PING latency and iPerf bandwidth test results from the most recent 1 minute. Based on the number of migration tasks currently being served by the source, the source with the highest overall score is selected as the source for this data acquisition. Preferably, if a source is located within the same data center, its network quality score will receive a significant boost, prioritizing "on-site data computation".

[0091] A three-layer verification system: 1) Transport layer verification: When transmitting data in chunks via HTTP / 3, each data block (e.g., 4MB) is accompanied by a CRC32 or SHA-1 checksum. The receiver verifies the checksum immediately upon receiving each block; if it fails, the block is retransmitted. 2) Object layer verification: After the entire object is transmitted, the receiver calculates its complete SHA-256 hash value and compares it with the checksum field in the declaration. Only if they match completely is the object marked as "ready". 3) Business layer verification: For files of specific formats (e.g., TensorFlow SavedModel), URFS can call a lightweight verification plugin to check whether the model format is correct, whether the input and output tensor dimensions meet expectations, etc.

[0092] The two-stage migration decision module, connected to the storage orchestration and verification module, performs pre-checks and assessments before data migration. These assessments include at least migration feasibility, estimated transmission time, and overall cost. During the execution phase, it uses a hash value generated from the object version, task ID, target center, and time window as an idempotent key to ensure the idempotency and recoverability of the migration operation. Specifically, it includes a pre-check unit and an execution unit. Wherein:

[0093] The pre-check unit is used to assess the data migration transmission time, overall cost, and risk score before the migration is executed. The pre-check unit is invoked during the scheduler's resource binding decision process, and its core function is to quickly simulate and comprehensively evaluate the proposed data migration scheme before physical resources are invested.

[0094] When the scheduler evaluates whether to schedule a task to a candidate data center (e.g., "Data Center B"), it synchronously calls the interface of the pre-check unit, passing in the list of data to be migrated and the target data center "B". This unit then initiates a virtual simulation, primarily evaluating three aspects:

[0095] 1) Transmission Time Estimation: The unit estimates transmission time based on the actual size of the data object, the current measured effective bandwidth from the source to the target center, the basic network latency, and the preset verification overhead, using the formula: Make accurate estimates.

[0096] 2) Comprehensive cost assessment: The unit combines the billing price of data transmission, the storage rental fee of the target center, and the opportunity cost that migration may bring, and conducts a quantitative analysis from a financial perspective.

[0097] 3) Risk Quantification Scoring: Unit queries real-time monitoring and historical databases to dynamically calculate risk scores. For example, if the target network's packet loss rate increases sharply recently, the score for the "Network Risk" sub-item will increase; if the target's storage space is critically low, the score for the "Storage Risk" sub-item will increase.

[0098] The pre-checking unit encapsulates the above evaluation results (feasibility, estimated time, cost, and risk score) and immediately feeds them back to the scheduler. The scheduler uses this result as a key negative cost factor and includes it in the overall cost of the candidate scheduling scheme. If the unit determines that the scheme is infeasible (e.g., insufficient storage space), the scheduler will directly exclude the option, avoiding invalid resource reservations and lengthy waiting timeouts.

[0099] The execution unit is used to perform the actual migration after the pre-check passes, and adopts an idempotency guarantee mechanism, including a hash value generated based on the object version, task ID, target center and time window as an idempotency key to uniquely identify the migration task and prevent duplicate execution; the execution unit also includes a status checkpoint mechanism and a conflict detection mechanism; the status checkpoint mechanism is used to record the status of key steps in the migration process and supports recovery from breakpoints; the conflict detection mechanism is used to detect and handle resource conflicts between concurrent migration tasks.

[0100] When the scheduler makes its final decision and triggers the actual data migration, the execution unit begins its work. This unit is responsible for safely and reliably implementing the pre-checked solution; its core design principle is to ensure the determinism of large-scale migration operations in a distributed environment. Specifically:

[0101] 1) Idempotency Guarantee Mechanism: To prevent duplicate operations caused by network retries, component restarts, etc., the execution unit generates a globally unique idempotency key for each migration task, which is constructed as follows:

[0102] Idempotent key = SHA-256(object version "commit-abc123" + task ID "job-001" + target center "dc-b" + time window "20231015-14");

[0103] Any request carrying the same idempotent key is treated as the same operation, ensuring that data is migrated only once.

[0104] 2) State Checkpoint Mechanism: The execution unit models each migration task as a state machine (e.g., "Initialization" → "Data Transmission in Progress" → "Data Verification in Progress" → "Completion"). At each critical node of the state transition, the unit persists the complete progress as a checkpoint. If the task is interrupted in the "Data Transmission in Progress" state, it can resume precisely from the previous checkpoint after recovery, without having to start from scratch, greatly improving fault tolerance and efficiency.

[0105] 3) Conflict Detection Mechanism: In high-concurrency scenarios, the execution unit can detect multiple requests from tasks to migrate the same data object to the same target. In this case, it will initiate conflict resolution logic, such as merging these requests into a single background migration task, allowing all initiators to share the execution result, thereby completely avoiding duplicate data transmission and saving network bandwidth and system I / O resources.

[0106] Through the intelligent evaluation of the pre-check unit and the reliable operation of the execution unit, this module ensures that cross-center data migration is both economical and prudent, as well as robust and efficient, forming an indispensable reliable data movement layer in the joint storage and computing scheduling.

[0107] The secure access verification module, connected to the storage orchestration and verification module, is used to generate a signature link based on cryptographic signatures, with access scope and time-limited restrictions. Based on this link, it implements multi-layered secure access and integrity verification of data objects, including HTTP header verification, range request verification, and end-to-end verification. Specifically, it includes:

[0108] 1) Signature link generation and distribution based on the principle of least privilege;

[0109] This module is invoked to generate a secure temporary access credential when a system process (such as the storage orchestration module or pre-check unit) needs to access a data object stored in another data center object store (such as S3 or OSS).

[0110] a. Credential Request and Link Construction: The requesting party (e.g., the scheduler located in data center A) first requests a temporary security token from the security token service of the data object's location center (e.g., data center B). This token has strictly limited attributes: a very short validity period (typically 5-15 minutes), access permissions only for specific buckets and object paths, and operation permissions limited to performing data reads and metadata queries (such as GET, HEAD). Based on this temporary token, this module uses the HMAC-SHA256 algorithm to generate a signed link. This link is an encrypted signed URL that encodes all security constraints, including the signature itself, expiration timestamp, issuer identity, allowed object keys, and allowed HTTP methods (such as GET).

[0111] b. Secure Distribution and Use: The generated signed link can be securely distributed to components that actually perform data access or verification operations, such as a lightweight verification agent within the target data center. This agent only holds this temporary link and has no access to the permanent storage access key. This design strictly adheres to the "principle of least privilege," ensuring that even if the link is accidentally leaked, the potential security risks are extremely limited due to its strict time, scope, and operational constraints.

[0112] 2) A multi-layered proactive verification process executed by the verification agent;

[0113] The verification agent holding the signed link is responsible for performing multi-layered integrity verification of the target data object. This verification process is proactive and non-intrusive, designed to uncover potential problems as much as possible without fully retrieving the data.

[0114] Layer 1: HTTP Header Metadata Validation: The validation proxy first sends an HTTP HEAD request to the address pointed to by the signed link. By analyzing the response, it can quickly verify several key pieces of information: the HTTP status code (confirming that the object exists and is accessible), the Content-Length header (confirming that the object size matches the declaration or expectation), and the Last-Modified and ETag headers (helping to determine if the object version may have changed). This layer of validation has extremely low cost, can be completed in milliseconds, and is used to quickly eliminate obvious inaccessibility or metadata mismatch issues.

[0115] Layer Two: Chunked Range Requests and Partial Content Verification: For objects requiring content verification, the proxy does not necessarily need to download all the data. It can logically divide the object into multiple contiguous ranges (e.g., 4MB each) and then initiate multiple HTTP GET requests with Range headers in parallel to retrieve the individual data blocks of the object. For each retrieved data block, the proxy independently calculates its hash digest. This layer of verification effectively detects potential partial corruption of data during transmission or storage, and achieves a good balance between verification depth and efficiency through parallelization and requesting only partial data.

[0116] Layer Three: Complete Object Acquisition and End-to-End Verification: For critical data, or when discrepancies are found in the second-layer verification, the agent can perform final end-to-end verification. This involves downloading the complete object content using a signed link, calculating its overall cryptographic hash (e.g., SHA-256), and rigorously comparing this calculated value with the trusted original checksum provided in the data claim. Only if they match perfectly is the object finally marked as verified. Furthermore, for specific data types, this stage can integrate business logic verification plugins to verify the correctness of its internal format and semantics.

[0117] By combining the temporary signature linking mechanism with the proactive multi-layer verification process, this module provides a powerful and auditable verification capability for the consistency and integrity of cross-center data while ensuring data access security and compliance, forming a key link in the trustworthy foundation of the entire system's data.

[0118] The in-memory computing joint scheduling module, connected to both the unified data declaration module and the two-stage migration decision module, optimizes the computation based on a joint cost function that integrates computation cost, data migration cost, cross-center communication cost, and data reuse benefits, according to the data dependency graph, resource status of each data center, and network conditions. It dynamically selects the optimal computing center and nodes and continuously learns and adjusts the weight parameters in the cost function based on historical execution data. Specific implementation includes:

[0119] 1) Design of joint cost function and dynamic cost assessment;

[0120] The fundamental basis for scheduling by the in-memory computing joint scheduling module is a carefully designed joint cost function. For a given computing task, it evaluates the total cost of scheduling it to a computing node N within a candidate data center C. , is defined by the following formula:

[0121]

[0122] Where α, β, γ, and δ are dynamically adjustable positive weighting coefficients. They represent the relative importance of computational cost, migration cost, cross-center cost, and reuse benefits in the overall decision-making process, respectively. The specific evaluation methods for each cost and benefit are as follows:

[0123] Calculate resource costs ( ): Evaluate the computational resource overhead of running the task on the target node N. This includes not only the node's current real-time load (CPU / memory utilization), but more importantly, the estimated queue waiting time for the task on this node (if resources are scarce), the deployment efficiency penalty due to resource fragmentation, and the performance degradation impact of the node's long-term load factor.

[0124] The cost of migrating data to the target center ( This assesses the cost of migrating all data required for the task from its current location to the target data center C. This cost is directly derived from the output of the pre-check unit in the two-stage migration decision module. It is quantified as follows:

[0125]

[0126] If the data already exists in the target center, this cost is zero.

[0127] Additional costs of cross-center communication ( This additional cost arises if the task requires frequent communication with services deployed in other data centers during execution (e.g., accessing remote databases or microservices). It is estimated based on the network topology distance between data centers, average communication latency, and potential cross-domain data compliance management costs.

[0128] Benefits arising from data already existing or reusable in the target center ( This is a negative cost (i.e., benefit), encouraging data locality and sharing. This benefit occurs if a usable copy of the data required by the task already exists in the target data center C (whether persistent storage or hot caching), or if the task's output data can be reused by other expected tasks within data center C. The size of the benefit is positively correlated with cache hit rate, computational savings from data sharing, and performance improvements from pure data locality.

[0129] 2) Optimization scheduling decision-making process based on cost function;

[0130] This module, as an extended scheduler plugin, is integrated into the scheduling framework of container orchestration platforms (such as Kubernetes). Its decision-making process is a multi-stage selection and optimization process:

[0131] a. Initialization and Information Collection: After receiving a Pod to be scheduled, the scheduler obtains its structured data dependency graph from the unified data declaration module. Simultaneously, it collects real-time resource status, network health status, and global data distribution index of all available data centers and nodes from the cluster monitoring system.

[0132] b. Pre-screening: Based on the hard constraints declared by the Pod (such as the requirement to use a GPU and minimum memory requirements) and data dependencies, candidate data centers and nodes that do not meet the conditions at all are quickly filtered out to form an initial candidate set.

[0133] c. Cost Calculation and Optimization Solution: For each combination of (data center C, compute node N) in the candidate set, calculate its joint cost J(C, N) in parallel. This process will trigger optimization. The evaluation (i.e., call pre-check) is then performed. Subsequently, the scheduler employs an efficient heuristic algorithm (such as a greedy algorithm with pruning or simulated annealing) to find the optimal solution that minimizes the total cost J among all candidate combinations. , ).

[0134] d. Final feasibility verification and binding: Before making the final decision, the scheduler will check the optimal solution ( , The scheduler requests a final, quick confirmation from the two-stage migration decision module (which can be considered a final verification of the pre-check). Only when the data migration plan is confirmed to be fully feasible and resources are available will the scheduler bind the Pod to the node. This triggers subsequent data migration and task startup processes.

[0135] 3) Online learning and adaptive adjustment of weight parameters;

[0136] To enable scheduling decisions to continuously adapt to dynamically changing cluster workloads and network environments, this module incorporates a weight learning unit to achieve online self-optimization of parameters α, β, γ, and δ.

[0137] The weight learning unit continuously collects data on historical scheduling events: at each scheduling decision, it records the predicted costs and the weights used; after the task is completed, it collects actual cost metrics (such as cost converted from actual runtime and actual cloud resource fees). The learning unit periodically (e.g., hourly) calculates the deviation between the predicted total cost and the actual total cost, and uses optimization algorithms such as gradient descent to automatically adjust the weight parameters used in the next cycle, aiming to minimize the mean squared error of this deviation.

[0138] Through this closed-loop feedback mechanism, the system can automatically learn how to weigh the importance of each cost factor in different scenarios (such as nighttime batch processing jobs and daytime online service peak periods), enabling the scheduling strategy to continuously evolve and thus maintain the optimal global resource utilization and cost efficiency in the long term.

[0139] This invention introduces a unified data declaration mechanism based on Kubernetes Pod annotations, standardizing and encapsulating the descriptions and strategies for three types of data (such as location, version, verification information, and proximity requirements). This enables the system to automatically parse and generate a structured data dependency graph, providing the scheduler with a global and accurate view of data dependencies. Consequently, operations personnel only need to master a simple annotation syntax to declare all data objects, eliminating the need for complex and error-prone configurations across multiple systems.

[0140] Furthermore, by employing a two-stage data migration decision-making mechanism, the system simulates and evaluates the feasibility, time, cost, and risks of the migration plan in the pre-check phase, and feeds the results back to the scheduling decision, avoiding scheduling tasks to centers where data migration is infeasible or extremely costly. In the execution phase, mechanisms such as idempotent keys, state checkpoints, and conflict merging ensure the reliability and efficiency of the migration operation.

[0141] The in-memory computing joint scheduling module employs a joint cost function that integrates computing costs, data migration costs, cross-center communication costs, and data reuse benefits. Before scheduling decisions, a migration pre-check is performed, and a heuristic algorithm is used to find the global optimum. The learning mechanism for weight parameters enables the system to dynamically adapt to environmental changes. Through pre-checks and intelligent scheduling, tasks are more likely to start on nodes where data is ready or where migration costs are lowest, significantly shortening the time from task submission to execution. The joint cost function comprehensively considers computing resource load, data migration costs, and cross-center communication overhead, dynamically guiding tasks to the resource combination with the lowest overall cost and highest efficiency. This reduces idle computing resources and data transmission bandwidth contention, improving the overall throughput and resource utilization of the cross-center cluster. By reducing unnecessary or inefficient cross-center data migration, network bandwidth usage costs are directly reduced. Simultaneously, higher resource utilization means that under the same business load, the required infrastructure resource scale may be reduced, thereby lowering the total cost of ownership (TCO).

[0142] By implementing a three-layer verification system in the storage orchestration and verification module—transport layer block verification, object layer end-to-end digest verification, and business layer semantic verification—combined with the proactive verification process of the secure access verification module, issues such as data corruption, version errors, or format mismatches can be promptly detected and corrected during data acquisition and migration. This prevents defective data from entering the computation stage, significantly reducing runtime task failures or incorrect results caused by data problems. The layered verification mechanism can quickly locate consistency issues at each stage of the data flow (during transmission, after storage, and before use), shifting the problem discovery time from reactive troubleshooting after task failure to the data preparation stage, greatly shortening the mean time to recovery (MTTR). Furthermore, every data migration, verification, and use leaves a complete and verifiable record, forming a clear data lineage map, enhancing the traceability and credibility of computation results, which is particularly important for scenarios with high data traceability requirements, such as AI training and scientific computing.

[0143] The secure access verification module generates encrypted signature links with strict temporal, spatial, and operational scope restrictions for use by third-party verification agents. These agents can perform multi-layered data verification without needing to know the permanent key. This reduces the risk of core access credentials being exposed during transmission and use, aligning with best security practices. Simultaneously, a standardized and auditable external data verification process is established, with detailed logging of all accesses via the signature links, providing a complete chain of evidence to meet data security and compliance audit requirements.

[0144] Example 2, as Figure 6 As shown, this invention also discloses a cross-center storage and computing joint scheduling method based on data proximity, comprising the following steps:

[0145] S1: Unified declaration and resolution of data dependencies;

[0146] Users submit computation tasks (such as an AI training job) using a standard Kubernetes Pod template. In the Pod's `metadata.annotations` field, users declare all data objects that the task depends on, using a unified format defined in this invention. This declaration must include at least three types of data objects: datasets, models, and code. For each object, it must specify its location identifier (e.g., `s3: / / bucket / data`), version information (e.g., `v1.2.0` or `commit-hash`), integrity checksum (e.g., SHA-256 value), and key scheduling policy parameters (e.g., `required` for data proximity, `migrate-then-pin` for storage, and `strict` for consistency level).

[0147] The unified data declaration module within the system (specifically an annotation adapter) listens for and captures the Pod. It parses the content of the `data.dependencies` annotation, transforming the scattered data descriptions into an internal structured data object model, and further constructs a data dependency graph in the form of a directed acyclic graph. This graph clarifies the dependencies and acquisition priorities between various data objects, providing a global and semantically rich view of data dependencies for subsequent scheduling.

[0148] S2: Optimal discovery and pre-checking of data sources;

[0149] Based on the data dependency graph generated in step S1, the system initiates a cross-center data source discovery process. The multi-source scanning unit in the storage orchestration and verification module sends queries in parallel to all participating data centers to obtain metadata (existence, size, location, etc.) for each data object replica. Subsequently, the availability assessment unit performs real-time status assessments on each available data source, including network quality (latency, bandwidth), current load, historical reliability, and transmission cost. The optimization decision unit applies a comprehensive scoring function (e.g., ...). From these, the optimal data source is selected.

[0150] Before the scheduler makes its final decision, the pre-check unit of the two-stage migration decision module is triggered. For each hypothetical scenario of "scheduling the task to a candidate data center," it simulates the required data migration and generates an evaluation report, including: estimated transmission time (based on object size and measured bandwidth), overall cost (transmission and storage costs), and risk score (based on network and storage health). This report serves as a key input to the scheduling decision process.

[0151] S3: Secure data migration and multi-layered integrity verification;

[0152] Once the scheduler makes a final decision, determines the target computing center and node for the task, and confirms that data migration is required, it enters the safe migration execution phase.

[0153] First, the secure access verification module generates an encrypted signature link (based on HMAC-SHA256) with a short validity period and strictly limited access objects and operation permissions, based on the storage location of the source data. This link is securely transmitted to a lightweight verification agent in the target data center.

[0154] The verification agent uses this signed link to perform proactive, multi-layered verification of data objects without exposing permanent credentials: 1) HTTP header verification: quickly verifying object metadata via HEAD requests; 2) Range request verification: efficiently detecting data corruption by retrieving data in chunks and calculating chunk digests via parallel Range GET requests; 3) End-to-end verification (for critical data): downloading the complete object and calculating the overall digest, then performing a final comparison with the verification value declared.

[0155] Meanwhile, the execution unit of the two-stage migration decision module is responsible for coordinating the actual transfer process. It generates a globally unique idempotent key (generated by hashing object version, task ID, target center, etc.) for this migration, ensuring that the operation is reentrant and does not repeat. The migration process is driven by a state machine and sets checkpoints to support recovery from breakpoints after failure, rather than starting from the beginning.

[0156] S4: Data proximity-aware joint storage and computing scheduling decision-making and task execution;

[0157] After filtering out candidate nodes that meet the basic resource constraints, the scheduler does not make a decision immediately. Instead, it calculates a joint cost function J for each candidate node (associated with its data center):

[0158] Among them, the cost of calculating resources ( ): Evaluate the overhead of running tasks on this node, including queue waiting time, resource fragmentation penalties, etc.

[0159] The cost of migrating data to the target center ( ): Directly adopt the evaluation results output by the pre-check unit in step S2 for the candidate solution.

[0160] Additional costs of cross-center communication ( If the task requires communication with other central services, additional network latency and compliance costs will apply.

[0161] Benefits arising from data already existing or being reusable in the target center ( If the data is already cached or shared in the target center, the corresponding revenue will be deducted to encourage data localization.

[0162] The scheduler uses a heuristic algorithm (such as a greedy algorithm) to select the (data center, node) combination that minimizes the total cost J as the optimal scheduling target. A final feasibility check is performed before final binding. After a task is scheduled to the target node, the required data is either ready (existing locally) or being efficiently and securely synchronized via step S3. The Kubelet on the node then pulls the image and starts the container, and the task begins execution.

[0163] S5: Continuous learning and optimization of weight parameters;

[0164] To enable the scheduling strategy to adapt to environmental changes, the system runs a continuous learning loop in the background. The weight learning unit of the in-memory computing joint scheduling module collects historical data: the predicted cost of each scheduling (based on the α, β, γ, δ at that time) and the actual cost after the task is completed (converted into resource costs and time costs).

[0165] The learning unit periodically analyzes the deviation between predictions and actual results and uses optimization algorithms such as gradient descent to automatically adjust the weight parameters (α, β, γ, δ) used in the next scheduling cycle. This allows the scheduling system to continuously evolve; for example, it can automatically increase β when network bandwidth costs rise. The weights of these weights are adjusted to maintain the economy and efficiency of scheduling decisions in the long term.

[0166] Through the close connection and closed-loop operation of the above five steps, the method of the present invention systematically realizes the joint intelligent scheduling of computing tasks and data flow in a complex cross-center environment, achieving the comprehensive goal of improving performance, reducing costs, and ensuring reliability and security.

[0167] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A cross-center storage and computing joint scheduling system based on data proximity, characterized in that: include: The unified data declaration module is used to receive and parse the data dependency information declared in Kubernetes Pod annotations. The annotations encapsulate the attributes and scheduling strategies of three types of data objects required by the task—dataset, model, and code—in a unified format, and generate a structured data dependency graph for scheduling decisions. The storage orchestration and verification module, connected to the unified data declaration module, is used to perform cross-data center data object discovery based on the data dependency graph through the unified remote file system (URFS), select the best option based on multi-dimensional scores of network quality, source load, reliability and cost, and implement multi-layer data consistency verification including at least transport layer block verification and object layer end-to-end verification. The two-stage migration decision module, connected to the storage orchestration and verification module, is used to perform a pre-check assessment before data migration. The assessment includes at least migration feasibility, estimated transmission time and overall cost. During the execution phase, the hash value generated by the object version, task ID, target center and time window is used as the idempotent key to ensure the idempotency and recoverability of the migration operation. The secure access verification module, connected to the storage orchestration and verification module, is used to generate a signature link based on cryptographic signatures with access scope and timeliness restrictions, and to implement multi-layered secure access and integrity verification of data objects based on this link, including HTTP header verification, range request verification and end-to-end verification. The storage-computing joint scheduling module is connected to the unified data declaration module and the two-stage migration decision module, respectively. It is used to optimize the solution based on the data dependency graph, the resource status of each data center and the network status, and a joint cost function that integrates computing cost, data migration cost, cross-center communication cost and data reuse benefits. It dynamically selects the optimal computing center and computing node, and continuously learns and adjusts the weight parameters in the cost function based on historical execution data. The joint cost function used by the in-memory computing joint scheduling module is: ； in, To calculate resource costs, including queue waiting time, resource fragmentation penalty, and node load factor; The cost of migrating data to the target center includes transmission time costs, verification overhead, and storage rental costs; Additional costs associated with cross-center communication include distance penalties, latency penalties, and compliance costs; The revenue generated from data existing or being reusable in the target center includes cache hit revenue, sharing revenue, and locality reward; α, β, γ, and δ are dynamically adjustable weighting coefficients.

2. The cross-center storage and computing joint scheduling system based on data proximity as described in claim 1, characterized in that: In the unified data declaration module, the data dependency information includes at least one of the following data objects: dataset, model, and code; each data object declares at least one of its location identifier, version information, size information, and verification summary through annotations. The annotation also includes strategy parameters, which include at least one of data proximity requirements, storage strategy, and consistency level. The unified data declaration module also includes an adapter component, which is used to parse the annotation and generate a structured data dependency graph. The data dependency graph is used to identify the dependencies between data objects, calculate the data acquisition priority, and evaluate the data access pattern.

3. The cross-center storage and computing joint scheduling system based on data proximity as described in claim 1, characterized in that: The unified remote file system includes: The multi-source scanning unit is used to query the object storage of each data center in parallel, collect object metadata, and build a global location index. The availability assessment unit is used to detect the network connectivity, historical access success rate, and response time of each storage source. The optimal decision-making unit is used to select the optimal source from multiple available data sources based on a scoring function. The verification execution unit is used to implement at least two of the following: transport layer block verification, object layer end-to-end verification, and service layer semantic verification.

4. The cross-center storage and computing joint scheduling system based on data proximity according to claim 3, characterized in that: The source selection scoring function used by the optimal decision-making unit is: ； in, Calculated based on round-trip time and available bandwidth; Calculated based on the current number of transmission tasks; Calculated based on historical access success rate; Calculated based on storage and transmission costs; , , , These are configurable weighting coefficients.

5. A cross-center storage and computing joint scheduling system based on data proximity as described in claim 1, characterized in that: The two-stage migration decision module includes: The pre-inspection unit is used to assess the data migration transfer time, overall cost, and risk score before performing the migration; The execution unit is used to perform the actual migration after the pre-check passes, and adopts an idempotency guarantee mechanism, including a hash value generated based on the object version, task ID, target center and time window as an idempotency key, which is used to uniquely identify the migration task and prevent duplicate execution. The execution unit also includes a state checkpoint mechanism and a conflict detection mechanism; the state checkpoint mechanism is used to record the state of key steps in the migration process and supports recovery from breakpoints; the conflict detection mechanism is used to detect and handle resource conflicts between concurrent migration tasks.

6. A cross-center storage and computing joint scheduling system based on data proximity according to claim 1, characterized in that: The signature link generated by the secure access verification module is an encrypted string based on the HMAC-SHA256 algorithm and contains the following information: signature value, expiration time, issuer, access scope, and allowed HTTP methods; The multi-layered security access and integrity verification performed by the secure access verification module includes: First-level verification: Obtain and verify object metadata based on HTTP HEAD requests; Second-level verification: Perform block-based verification on the object based on the HTTP Range request; Third-level verification: Download the complete object and perform end-to-end digest verification.

7. A cross-center storage and computing joint scheduling system based on data proximity according to claim 1, characterized in that: The in-memory computing joint scheduling module also includes a weight learning unit, which is used to collect historical scheduling decisions and execution results, and dynamically update the weight coefficients α, β, γ, and δ using the gradient descent method.

8. A cross-center storage and computing joint scheduling method based on data proximity, applied to the system as described in any one of claims 1-7, characterized in that: Includes the following steps: S1: Unified Declaration and Resolution of Data Dependencies: Receives Kubernetes Pod annotations carrying data dependency declarations. These declarations encapsulate the location, version, verification information, and scheduling strategy of three types of data objects required by the task: dataset, model, and code, in a unified format. Parses the annotations and generates a structured data dependency graph for the scheduler to globally evaluate data dependencies. S2: Data source selection and pre-check: Based on the data dependency graph, multiple data centers are scanned in parallel through a unified remote file system to discover available copies of each data object; The data source is selected from multiple replicas based on a comprehensive scoring function, which considers at least network quality, source load, and transmission cost. Before triggering the actual data migration, a pre-check is performed to assess the feasibility of the migration, estimate the transmission time, and the overall cost. S3: Secure Migration and Integrity Verification: If the pre-check passes, an encrypted signature link with access scope and time limits is generated; data migration is performed based on this link, and multi-layer integrity verification including chunked transmission verification and end-to-end digest verification is implemented during the migration process; S4: Joint scheduling and task execution of storage and computing: Based on the data dependency graph, pre-check evaluation results, real-time resource status and network conditions of each data center, construct and solve the joint cost function, and use the joint cost function to calculate the total cost of each candidate scheduling; select the computing center and computing node with the optimal total cost according to the solution results; finally, schedule the computing task to the selected node for execution.

9. The cross-center storage and computing joint scheduling method according to claim 8, characterized in that: The optimal selection and pre-check in step S2 specifically include: S201: Multi-Source Discovery and Scoring: Parallel queries of object storage in each data center to obtain metadata, version, and location information of data objects; for each available data source, its source selection score is calculated using the following formula: ； in, Calculated based on round-trip time and available bandwidth; Calculated based on the current number of transmission tasks; Calculated based on historical access success rate; Calculated based on storage and transmission costs; , , , These are configurable weighting coefficients; S202: Three-layer verification mechanism: Three-layer consistency verification is performed during data transmission, including: real-time digest verification of each data block at the transport layer; end-to-end verification of the complete object using MD5 or SHA256 at the object layer; and semantic compliance verification based on data format and dependencies at the business layer. S203: Two-stage decision-making: In the pre-check phase, calculate: ； The migration risk score is assessed; the execution phase is initiated only when the estimated time and risk are below the threshold. During the execution phase, a hash value generated from the object version, task ID, target center, and time window is used as an idempotent key to ensure the uniqueness and recoverability of the migration operation.

10. The cross-center storage and computing joint scheduling method according to claim 8, characterized in that: The in-memory computing joint scheduling decision in step S4 specifically includes: S401: Define the joint cost function as follows: ； in, To calculate resource costs, including queue waiting time, resource fragmentation penalty, and node load factor; The cost of migrating data to the target center includes transmission time costs, verification overhead, and storage rental costs; Additional costs associated with cross-center communication include distance penalties, latency penalties, and compliance costs; The benefits are those generated because the data already exists or can be reused in the target center, including cache hit benefits, sharing benefits, and locality rewards; α, β, γ, and δ are dynamically adjustable weighting coefficients. S402: Dynamic Parameter Learning: Collect historical scheduling decisions and their corresponding actual task execution performance data, calculate the deviation between predicted costs and actual costs; use gradient descent to dynamically update and optimize the weight parameters α, β, γ, δ so that the scheduling decisions continuously adapt to changes in cluster load and network environment.

Images