File fragment uploading and dynamic routing method and system for multi-cloud heterogeneous environment

Abstract

The application discloses a file fragment uploading and dynamic routing method and system for a multi-cloud heterogeneous environment, belongs to the technical field of cloud storage, and comprises the following steps: S1, infrastructure and service definition; S2, dynamic routing strategy configuration; S3, intelligent routing matching and decision-making; S4, uploading process control and fragment strategy; and S5, execution monitoring, merging and ending. Through the above manner, the application constructs a unified bottom abstract architecture, uniformly encapsulates the configuration, interface and service of different cloud storage platforms, and realizes the quick access of various heterogeneous storage services by adopting an inheritance expansion mechanism, thereby effectively shielding the implementation differences between different cloud platforms, reducing the system coupling degree, improving the code reuse rate and system scalability.

Description

Technical Field

[0001] This invention relates to the field of cloud storage technology, specifically to a method and system for file fragment uploading and dynamic routing in multi-cloud heterogeneous environments. Background Technology

[0002] With the continuous development of cloud computing, distributed storage, and object storage technologies, more and more business systems are adopting cloud storage platforms for file storage and management to meet the needs of massive data storage, high-concurrency access, and cross-regional resource sharing. Meanwhile, to balance storage costs, business performance, data security, and resource reliability, enterprises are gradually adopting multi-cloud deployment models, storing data from different business operations across object storage services provided by multiple cloud vendors. This fully leverages the performance advantages of different cloud storage platforms and improves overall resource utilization. Therefore, file upload technology for multi-cloud heterogeneous environments is gradually becoming an important component of cloud storage system construction, with broad application prospects in areas such as internet platforms, government systems, enterprise information systems, and industrial data management.

[0003] Existing multi-cloud file upload solutions typically encapsulate the basic operations of different cloud storage platforms through a unified interface and select the target storage service based on pre-configured business rules. Upon receiving a file upload request, the system first determines the corresponding storage node based on the business type or configuration strategy. Small files are usually uploaded directly, while large files are uploaded in chunks to complete the data transmission. After all chunks are uploaded, the storage service merges the files and returns the final file access address.

[0004] Existing multi-cloud file upload solutions still have certain limitations. On the one hand, there are still significant differences in interfaces and configurations between different cloud storage platforms. When integrating new storage services, a large amount of adaptation and development work is usually required, resulting in insufficient system scalability and high maintenance costs. On the other hand, the selection of storage nodes in the current upload process mostly relies on fixed configurations or static mapping methods, making it difficult to comprehensively consider operational status information such as network latency, storage costs, node reliability, and security levels. It is also impossible to dynamically select the optimal upload path according to different business scenarios, and it is difficult to complete intelligent switching in a timely manner when the target node fails, thus affecting upload efficiency and system stability. Summary of the Invention

[0005] To address the aforementioned shortcomings of existing technologies, this invention provides a method and system for file fragmentation upload and dynamic routing in multi-cloud heterogeneous environments.

[0006] To achieve the above objectives, the present invention provides the following technical solution: A method for file fragment upload and dynamic routing in multi-cloud heterogeneous environments includes the following steps: S1. Construct an abstract infrastructure; S2. Based on the abstract infrastructure, generate a mapping table and obtain a set of candidate storage service instances; S3. Based on the candidate storage service instance set, obtain the target storage service instance, which includes the following steps: S301. Construct the original feature vector based on the candidate storage service instance set, and use the extreme value normalization method to obtain the standardized feature vector; S302. Based on standardized feature vectors and combined with the priority configuration of the current business scenario, calculate the comprehensive score result of the candidate storage nodes; S303. Based on the comprehensive scoring results, perform fault circuit breaking and automatic switching, and output the routing results after circuit breaking; S304. Based on the routing results after the circuit breaker is triggered, perform fragmented upload initialization and concurrent upload, and finally output the target storage service instance; S4. Calculate the sharding strategy using the target storage service instance to obtain a sharded upload session. This includes the following steps: S401. Read and output target storage service information through the target storage service instance; S402. Based on the file size threshold in the target storage service information, perform adaptive sharding strategy calculation, obtain the shard size, and generate a shard index list; S403. Based on the fragment index list, perform fragment upload initialization and concurrent upload, and obtain the fragment upload session; S5. Through the fragmented upload session, calculate the fragmented status information, ordered merging vouchers, the final target file object, and the unified encapsulation adaptation result, and output the standardized file upload result and the full-link audit log.

[0007] Furthermore, in step S1, the abstract infrastructure specifically includes: standard business interfaces, standard configuration specifications, standard basic service capabilities, and highly cohesive infrastructure.

[0008] Furthermore, in step S1, the high-cohesion infrastructure specifically refers to: 1) Initiate business requests through the top-level cloud-native application, calculate the unified access path for the business requests, and obtain the unified access point; 2) By using a unified access point, the unified encapsulation result of request parameters, interface protocols, and access rules is calculated to obtain standardized business requests; 3) Receive standardized business requests through the load balancer configurator, calculate traffic distribution rules, task scheduling rules, and storage node matching rules, and obtain the request distribution results; 4) Based on the request distribution results, calculate the task bearing relationship between the metadata server, Hypertext Transfer Protocol node, standard protocol node, multi-cloud object storage instance and horizontally scaled object storage node to obtain the parallel heterogeneous service and storage medium layer; 5) Through parallel heterogeneous service and storage media layers, the aggregation relationship between each heterogeneous node and the underlying storage resources is calculated to obtain the object pool; 6) By using an object pool to uniformly aggregate physical storage media, cloud storage media, and multi-protocol storage nodes, the logical unity relationship of the underlying storage resources is calculated, resulting in a logically unified set of storage resources for physical and cloud storage media.

[0009] Furthermore, in step S2, by inheriting the standard configuration specifications and standard basic service capabilities in the pre-abstract infrastructure, third-party storage services are obtained. Based on the third-party storage services and combined with preset business scenario tags, a mapping relationship table between business scenarios and third-party storage services is calculated. According to the comparison between the set business scenario identifier and the mapping relationship table, the uploaded resource category is identified, and finally a candidate storage service instance set is output.

[0010] Furthermore, the original feature vector in step S301 includes: storage cost coefficient, network latency coefficient, service reliability coefficient, and security compliance level.

[0011] Furthermore, in step S302, the formula for calculating the comprehensive score result is as follows: ; in, This is the overall score result for the candidate storage nodes. These are the eigenvalues ​​of the normalized storage cost coefficient. These are the eigenvalues ​​of the normalized network delay coefficients. These are the characteristic values ​​of the normalized service reliability coefficient. These are the characteristic values ​​of the normalized security compliance level. The routing weights of the candidate storage nodes. These are the dynamic weighting coefficients corresponding to each evaluation indicator.

[0012] Furthermore, in step S402, the adaptive sharding strategy calculation specifically involves: By determining whether the size of the file to be uploaded exceeds a preset segmentation threshold, it is decided whether the current file should be uploaded in normal direct mode or in segmented mode. If the file size does not exceed the fragmentation threshold, the normal upload interface of the target storage service instance is called directly to upload the entire file. If the file size exceeds the fragmentation threshold, an adaptive fragmentation strategy is further calculated based on the real-time network bandwidth estimation results, and operations such as task initialization, context establishment, concurrent segmentation, data integrity verification, request encapsulation, and dynamic forwarding are performed.

[0013] Furthermore, in step S403, the chunked upload session includes upload task ID, file metadata, total number of chunks, chunk index list, uploaded chunk status, chunk verification information, target storage node information, and chunked upload request packet.

[0014] Furthermore, in step S5, based on the fragmented upload session, the monitoring and abnormal backoff retry of the fragmented upload results are calculated, and real-time fragmented status information is output. Input the real-time fragmentation status information. If there are still fragments that have not been completed, return to the adaptive fragmentation strategy calculation stage to continue processing the remaining fragments. If all fragments have been uploaded, collect the entity tag list and fragment sequence number of all successfully fragmented fragments, and arrange them according to the fragmentation order to obtain an ordered merge certificate. Based on the ordered merge certificate, a complete fragment upload request message body is constructed, and a complete fragment upload instruction is sent to trigger the merging operation of each fragment data to generate the final target file object; Based on the final target file object, calculate the final file access path, file size, hash value, storage location and complete metadata information returned by the storage service, as well as the adaptation and unified encapsulation of the response format returned by different storage protocols, and finally obtain a unified encapsulation and adaptation result.

[0015] A file fragment upload and dynamic routing system for multi-cloud heterogeneous environments, the system comprising: Infrastructure and Service Definition Module: Standard business interfaces, standard configuration specifications, standard basic service capabilities, and highly cohesive infrastructure to build an abstract infrastructure; Dynamic routing strategy configuration module: Based on the abstract infrastructure, it generates a mapping table and obtains candidate storage service instances; Intelligent routing matching and decision-making module: Constructs an original feature vector based on the candidate storage service instance set, uses the extreme value normalization method to obtain a standardized feature vector, and calculates the comprehensive score result of the candidate storage nodes by combining the standardized feature vector with the priority configuration of the current business scenario. Based on the comprehensive score result, it performs fault circuit breaking and automatic switching, outputs the routing result after circuit breaking, and finally performs fragmented upload initialization and concurrent upload based on the routing result after circuit breaking, outputting the target storage service instance. The process control and fragmentation strategy module reads and outputs target storage service information through the target storage service instance, performs adaptive fragmentation strategy calculation based on the file size threshold in the target storage service information, obtains the fragment size, generates a fragmentation index list, performs fragmentation upload initialization and concurrent upload based on the fragmentation index list, and obtains the fragmentation upload session. The execution closing module takes the input of the fragmented upload session, first extracts the real-time fragmented status information, then verifies the collected orderly merging vouchers, then triggers the server to merge and generate the final target file object, and finally outputs the standardized file upload result and full-link audit log delivered to the business layer.

[0016] Compared with the prior art, the beneficial effects of this invention are as follows: 1. By constructing a unified underlying abstract architecture, the configuration, interface and service of different cloud storage platforms are uniformly encapsulated, and the inheritance extension mechanism is adopted to realize the rapid access of various heterogeneous storage services. This effectively shields the implementation differences between different cloud platforms, reduces the degree of system coupling, improves code reusability and system scalability, and reduces the amount of subsequent adaptation development and maintenance costs. 2. At the same time, by establishing a mapping relationship between business scenarios and storage resources, and combining dynamic routing strategies to comprehensively evaluate the cost, network latency, reliability and security level of each storage node, intelligent selection of upload target nodes and automatic fault switching are achieved, thereby further improving the file upload efficiency, file upload stability, reliability and resource utilization efficiency in a multi-cloud environment. 3. This invention introduces a network-state-based adaptive fragmentation upload mechanism, dynamically calculating the fragment size and number based on real-time network bandwidth. Combined with a comprehensive control mechanism encompassing concurrent uploads, data integrity verification, persistent upload progress, resume uploads, retrying in case of errors, automatic fragment merging, and unified result encapsulation, this invention achieves dynamic optimization and unified management of the file upload process. Compared to traditional fixed fragmentation and static upload methods, this invention effectively reduces transmission performance fluctuations caused by changes in the network environment, minimizes retransmission overhead after upload failures, improves file upload efficiency, transmission reliability, and system compatibility, and further enhances the continuity, maintainability, and overall performance of file upload services in multi-cloud heterogeneous environments. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are merely some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without any creative effort.

[0018] Figure 1This is a schematic diagram of the overall process structure of the present invention; Figure 2 This is a schematic diagram of the high-cohesion basic architecture of the present invention; Figure 3 This is a schematic diagram illustrating the process of extending the inheritance base class method of the present invention. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0020] Example 1: As Figure 1-2 As shown, the file fragment upload and dynamic routing method for multi-cloud heterogeneous environments includes the following steps: S1: Infrastructure and Service Definition By constructing standard business interfaces, standard configuration specifications, standard basic service capabilities, and a highly cohesive infrastructure, we clarify the unified access rules, configuration constraint rules, service call rules, and underlying logic reuse rules in a multi-cloud heterogeneous storage environment, thus forming an abstract infrastructure.

[0021] The standard business interfaces include unified methods for service initialization, network probe, server connection establishment, asynchronous upload, synchronous upload, download, deletion, copying, and renaming. The standard configuration specification is used to unify the configuration methods of different storage services. The configuration classes corresponding to each storage service, including FtpConfig, OssConfig, MinioConfig, ObsConfig, TosConfig, etc., all inherit from the basic configuration classes or basic configuration interfaces such as BaseConfig and IBaseConfig. When adding a third-party storage service, the third-party storage service must follow the unified access rules, inherit the FileServiceBase base class and the IFileServiceBase base interface, and implement standard business methods such as service initialization, network probe, server connection establishment, asynchronous upload, synchronous upload, download, deletion, copying and renaming; Meanwhile, its service configuration class needs to inherit basic configuration classes or basic configuration interfaces such as BaseConfig and IBaseConfig to ensure consistency in access methods, configuration structures and calling processes for different storage services; Furthermore, the abstract infrastructure specifically refers to: (1) By extracting common file operation features such as uploading, downloading, updating, and deleting, the common file operation capabilities between different cloud storage platforms are calculated to obtain standard business interfaces; (2) By defining abstract configuration classes such as basic setting categories, basic setting interfaces and basic auxiliary categories, the configuration constraint relationships between different cloud vendors, different protocols and different storage media are calculated to obtain standard configuration specifications; (3) By defining custom abstract base classes, custom interfaces and other abstract service classes, calculate the calling rules of core service capabilities such as file upload, file download, file deletion, file update, file renaming and chunked upload, and obtain standard basic service capabilities; (4) By encapsulating, inheriting and reusing the underlying core logic through the basic abstraction method, the common execution logic between different storage service instances is calculated to obtain a highly cohesive infrastructure; Furthermore, such as Figure 2 As shown, a high-cohesion infrastructure is specifically as follows: 1) Initiate business requests through the top-level cloud-native application or distributed system, calculate the unified access path of the business requests, and obtain the unified access point (the request entry point that is received in a standardized manner by the RESTful API gateway or SDK). 2) By using a unified access point, the unified encapsulation result of request parameters, interface protocols, and access rules is calculated to obtain standardized business requests; 3) Receive standardized business requests through the load balancer configurator, calculate traffic distribution rules, task scheduling rules, and storage node matching rules, and obtain the request distribution results; 4) Based on the request distribution results, calculate the task carrying relationship between the metadata server, Hypertext Transfer Protocol (HTTP) node, standard (FTP) protocol node, multi-cloud object storage instance and horizontally scaled object storage node to obtain the parallel heterogeneous service and storage media layer; 5) Through parallel heterogeneous service and storage media layers, the aggregation relationship between each heterogeneous node and the underlying storage resources is calculated to obtain the object pool; 6) By using an object pool, physical storage media, cloud storage media, and multi-protocol storage nodes are aggregated in a unified manner, and the logical unity relationship of the underlying storage resources is calculated to obtain a logically unified set of storage resources for physical and cloud storage media; By using the above methods, cross-cloud vendor configuration structures, heterogeneous network protocol access rules, storage service call rules, and underlying resource aggregation rules are calculated to obtain an abstract infrastructure with high cohesion, high scalability, low coupling, and rapid adaptability. This reduces the system integration and management complexity when heterogeneous network protocols coexist, and solves the problem of rapid access and adaptation of storage services in a multi-cloud environment. The abstract infrastructure architecture achieves high cohesion and high scalability of the entire system architecture, standardizes the configuration structure across cloud vendors, significantly reduces the complexity of system integration and management when heterogeneous network protocols coexist, and fundamentally solves the problem of rapid access and adaptation in multi-cloud environments. Furthermore, the system is a macro-level overview of the overall framework, specifically referring to the overall architecture of the file fragment upload and dynamic routing system for multi-cloud heterogeneous environments; S2: Dynamic Routing Policy Configuration Based on the abstract infrastructure, a mapping table is generated to obtain a set of candidate storage service instances, specifically including the following steps: S201. Obtain third-party storage services by inheriting the standard configuration specifications and standard basic service capabilities from the front-end abstract infrastructure. This includes the following steps: (1) Third-party storage service object definition: Define the specific object storage type based on the idempotency principle; By combining standard configuration specifications and standard basic service capabilities, different cloud vendors, different protocols, or different storage media can access the system in the form of a unified object, avoiding data inconsistency problems caused by repeated initialization, repeated registration, or repeated calls; (2) Third-party storage configuration inheritance: Inherit standard configuration specifications and extend private parameter configurations for specific cloud vendors (such as AWS S3, Aliyun OSS) or storage service types; Private parameters include, but are not limited to, access keys, bucket names, access domain names, region identifiers, protocol types, encryption methods, and vendor-specific connection parameters, thereby maintaining a unified configuration standard while being compatible with the differentiated requirements of different vendors; (3) Physical storage method implementation: Inheriting standard basic service capabilities and implementing specific business logic, specifically covering core capabilities such as storage service initialization, ordinary file upload, fragmented file upload, streaming download, file deletion, file update, and file renaming, enabling different third-party storage services to be called by business systems under standard business interfaces, and extending the process as follows: Figure 2 As shown; S202. Based on third-party storage services and combined with preset business scenario tags, calculate the mapping relationship table between business scenarios and third-party storage services: (1) Definition of dynamic routing mechanism: By abstracting the highly cohesive infrastructure in the infrastructure, the core rules of dynamic routing are defined, and the classification standards for resource upload under different business scenarios are clarified; This enables the system to determine candidate storage ranges based on factors such as business type, file source, file purpose, access frequency, and security requirements. (2) Business scenario modeling: Preset multiple business scenario tags, including but not limited to PC files, mobile resources, electronic signatures, report data, personnel avatars, etc., and select the corresponding storage service type according to the access speed, cost, security, reliability and compliance requirements of different business scenarios; (3) Storage mapping configuration: Establish a mapping relationship table between business scenarios and third-party storage services, and initialize the corresponding storage service instance according to the mapping result, so that business requests can be accurately landed from the abstract scenario to the specific storage node; S203. Based on the comparison between the set business scenario identifier and the mapping relationship table, identify the uploaded resource category and finally output a set of candidate storage service instances; The candidate storage service instance set transforms the standard business interfaces, standard configuration specifications, and standard basic service capabilities defined in the abstract infrastructure into multi-cloud storage service instances that can be actually invoked, enabling the system to achieve unified access, unified configuration, unified invocation, and unified management in different storage service or protocol environments such as Alibaba Cloud OSS, Tencent Cloud COS, Huawei Cloud Obs, Volcano Cloud Tos, Minio, AWS S3, NFS, SMB, and HTTP. Through this step, the system no longer relies on a single cloud vendor or a single storage protocol. Instead, it can flexibly mount, initialize, and switch different third-party storage services according to business scenarios. At the same time, through business scenario tags and storage mapping relationship tables, the complex underlying storage differences are encapsulated within the system, shielding upper-layer business applications from the differences between different vendors, protocols, and storage media, thereby improving the system's scalability, maintainability, and cross-cloud adaptability. S3: Intelligent Routing Matching and Decision Making Based on the set of candidate storage service instances, the target storage service instance is obtained, which includes the following steps: S301. Construct the original feature vector based on the candidate storage service instance set, and obtain the standardized feature vector using the extreme value normalization method, specifically: Construct the original feature vector: 1) Storage cost coefficient (Cost): Used to characterize the differences in storage costs between different storage types or different cloud providers (e.g., standard storage is 1.0, infrequent storage is 0.8, and archive storage is 0.5). 2) Network latency coefficient (Latency): Obtained by sending heartbeat packets to each storage node through the gateway and calculating the average response time (ms); 3) Service reliability coefficient (Reliability): Calculated based on the HTTP 5xx error rate or request success rate within a preset time window; 4) Safety compliance level (Security): Match and score requirements such as encrypted transmission, regional compliance, and cross-regional access based on business scenarios; Furthermore, to eliminate differences in different indicator units (such as amount, time, percentage), an extreme value normalization method is used to map each feature component to the [0,1] interval, forming a standardized feature vector. The expression for the standardized feature vector is:

[0022] in, To standardize the feature vector, These are the eigenvalues ​​of the normalized storage cost coefficient. These are the eigenvalues ​​of the normalized network delay coefficients. These are the characteristic values ​​of the normalized service reliability coefficient. The characteristic values ​​of the normalized security compliance level are shown in Table 1.

[0023] S302. Based on standardized feature vectors and combined with the priority configuration of the current business scenario, calculate the comprehensive score result of the candidate storage nodes; Based on the feature vectors, the routing engine calls the scoring function according to the priority configuration of the current business scenario (such as "cost priority" or "speed priority"). Calculate the overall score for each candidate storage node:

[0024] in, This is the overall score result for the candidate storage nodes. These are the eigenvalues ​​of the normalized storage cost coefficient. These are the eigenvalues ​​of the normalized network delay coefficients. These are the characteristic values ​​of the normalized service reliability coefficient. These are the characteristic values ​​of the normalized security compliance level. Routing weights for candidate storage nodes , The dynamic weight coefficients corresponding to each evaluation indicator (satisfying) Corresponding storage cost weight, Corresponding network latency weights Corresponding service reliability weights (Corresponding security compliance level weights) Furthermore, dynamic weight configuration can include the following two typical modes: Speed-First Mode: If business scenario A is "mobile images", prioritize user access speed and reduce the priority of cost factors; the system will automatically increase the priority. (Network latency weight);

[0025] in, The highest percentage indicates that the system prioritizes storage services with better network link quality and shorter response time; Cost-first mode: If business scenario B is "historical report archiving", under the premise of ensuring basic availability, the cheapest storage is selected first, and the system automatically adjusts the cost. (Storage cost weight);

[0026] in, The highest percentage indicates that the system prioritizes storage nodes with lower prices or better storage costs; S303. Based on the comprehensive scoring results, perform fault circuit breaking and automatic switching, and output the routing results after circuit breaking; Based on the comprehensive score and routing weight of the candidate storage nodes, the routing policy engine locks the candidate storage node with the highest score as the current target storage node. The dynamic routing gateway will start a bypass health check to continuously monitor the connection status, response time and anomaly rate of the target node. If the target node is detected in Appearing consecutively within seconds If the second connection times out, the circuit breaker mechanism will be triggered, and the route weight will be automatically adjusted. Set the score to 0, recalculate the score, obtain the routing results after the circuit breaker is triggered, and seamlessly switch the traffic to the storage node with the second highest score. Furthermore, the test extended the storage services to include 8 types, with each type of storage service being called over 100 times, achieving a success rate of 99.9%. S304. Based on the routing results after the circuit breaker is triggered, perform fragmented upload initialization and concurrent upload, and finally output the target storage service instance; Based on the routing results obtained after circuit breaker correction, the system dynamically loads the corresponding storage service implementation class and reads the configuration parameters corresponding to the storage service, including access key, access domain name, storage bucket name, region identifier, protocol type, encryption method and connection timeout. The system initializes the target storage service instance based on the loaded configuration parameters, completes connection establishment, permission verification, protocol adaptation, client object creation, and runtime status checks, and puts the target storage service instance into an executable state. After the target storage service instance is initialized, the system uses the instance as the execution object of the current upload request and passes it to the subsequent upload process control stage so that it can continue to perform operations such as upload mode determination, direct upload of small files, uploading of large files in chunks, resume upload from breakpoint, chunk merging and result encapsulation. S4: Upload Process Control and Fragmentation Strategy By calculating the sharding strategy through the target storage service instance, a sharding upload session is obtained, which includes the following steps: S401. Read and output target storage service information through the target storage service instance: After completing intelligent route matching and initializing the corresponding storage service instance, the system starts the target storage service instance and enters the upload mode selection stage, and reads the target storage service information such as the file size, file type, and business scenario identifier of the file to be uploaded. S402. Based on the file size threshold in the target storage service information, perform adaptive sharding strategy calculation to obtain the optimal shard size and generate a sharding index list, specifically: By detecting whether the size of the file to be uploaded exceeds a preset chunking threshold (e.g., 100MB), it is determined whether the current file should be uploaded in normal direct upload mode or in chunked upload mode. For small files (files smaller than or equal to the threshold), the system directly calls the normal upload interface of the target storage service instance to perform a full upload; For large files, the system further performs adaptive fragmentation strategy calculation based on real-time network bandwidth estimation results, and achieves efficient and stable uploading of large files through operations such as task initialization, context establishment, concurrent segmentation, data integrity verification, request encapsulation and dynamic forwarding; Furthermore, the adaptive sharding strategy is calculated as follows: 1) Perform network probe speed test: Before fragmented upload initialization, the client sends a probe data packet of a preset size (e.g., 64KB) to the gateway, records the round-trip time (RTT) and transmission time, and estimates the effective bandwidth of the current uplink. ; 2) Dynamically calculate the optimal fragmentation threshold: based on the estimated bandwidth. and the preset number of concurrent threads Calculate the optimal partition size ; High bandwidth scenarios To avoid excessive CPU consumption from frequent HTTP connections established by small fragments, the system increases the fragment granularity. The optimal fragment size calculation formula is as follows:

[0027] Low bandwidth / unstable scenarios To reduce the retransmission cost after a single fragment fails, the system reduces the fragment granularity. The optimal fragment size is then calculated using the following formula:

[0028] in, To achieve the optimal fragment size, The maximum size of a data segment is limited by the cloud provider (e.g., 5GB). For the estimated bandwidth, Expected single-fragment upload time (e.g., 2 seconds). Minimum fragment size limit (e.g., 100KB); For medium bandwidth scenarios, the system can be configured based on default policies, historical upload performance, or business scenarios. and Choose an appropriate chunk size to balance upload efficiency and retransmission cost in case of failure; 3) Calculate the total number of fragments: based on the calculated dynamic optimal fragment size. Regarding the total file size Perform integer division to determine the final total number of fragments. And generate a list of shard indexes (each shard index records the shard number, start offset, end offset and data length, providing a basis for subsequent concurrent sharding and breakpoint resume). S403. Based on the fragment index list, perform fragment upload initialization and concurrent upload, and obtain the fragment upload session, specifically: Using the shard index list, the system first sends a shard upload initialization request to the target storage service to obtain a globally unique upload task ID (UploadId, which is used to identify this shard upload task and is continuously used in subsequent shard uploads, status recording, breakpoint resume, and shard merging processes). After obtaining the globally unique upload task ID, the system establishes a chunked upload context environment and persistently records file metadata, upload task ID, total number of chunks, chunk index list, uploaded chunk status, chunk verification information, and target storage node information. With this context, even if the upload process is interrupted, the system can continue to perform breakpoint resume based on the saved fragment state, avoiding duplicate uploads of successfully uploaded fragment data. Subsequently, the system starts a thread pool or coroutine pool, and performs concurrent reading and splitting of file data according to the splitting range calculated by the adaptive splitting strategy. During the file reading process, the system can use memory-mapped file technology or zero-copy mechanism to read the file stream, reduce the data copying overhead between kernel mode and user mode, and improve the efficiency of reading and uploading large files. The system adopts a producer-consumer model. While reading the file stream and splitting the data into segments, it asynchronously calculates the hash value (such as MD5 or CRC32) of each segment to generate the corresponding digital fingerprint. By decoupling the verification calculation from the network transmission process, it can prevent CPU-intensive verification operations from blocking I / O transmission, while ensuring the integrity and verifiability of each segment of data during transmission. The system then encapsulates the fragmented data, fragmented sequence number, upload task ID, fragmented offset, fragmented size, and corresponding verification information into a fragmented upload request packet, and sends it to the dynamic routing gateway via HTTP or HTTPS protocol. The request packet can also carry a business scenario identifier, a target storage node identifier, and an upload context identifier, so that the gateway can perform subsequent forwarding and status recording. After receiving a fragment upload request, the dynamic routing gateway forwards the fragment request to a specific cloud storage node based on the current network link quality, the load of the backend storage node, and the aforementioned intelligent routing decision results. When the network link or target node status changes, the gateway can adjust subsequent fragment requests in conjunction with the dynamic routing strategy, thereby improving the stability and continuity of the fragment upload process. Through the above processing, the system forms a chunked upload session. The chunked upload session includes UploadId, file metadata, total number of chunks, chunk index list, uploaded chunk status, chunk verification information, target storage node information, and chunked upload request packet. This chunked upload session is used as the input to step S5 for subsequent chunked result monitoring, exception retry, upload progress persistence, chunk merging, and resource cleanup. This step enables the system to quickly complete ordinary direct uploads for small files and achieve adaptive file fragmentation, concurrent uploads, and dynamic link forwarding for large files, thereby significantly improving the efficiency, stability, and resource utilization of file uploads in multi-cloud heterogeneous environments. S5: Execution monitoring, merging, and cleanup The process begins by inputting a fragmented upload session. First, real-time fragmented status information is extracted. Then, the collected ordered merging credentials are verified. Next, the server is triggered to merge and generate the final target file object. Finally, a standardized file upload result and end-to-end audit logs are delivered to the business layer. The process includes the following steps: S501. Based on the fragmented upload session, calculate the monitoring and abnormal backoff retry of the fragmented upload results, and output real-time fragmented status information. Based on the chunked upload session, the system receives the chunked upload result returned by the storage service. If successful, it records the entity tag (ETag) identifier and sequence number of the chunk as real-time chunked status information. If a segment upload fails, the exponential backoff retry mechanism is triggered to re-upload the segment. The segment upload is re-executed within the retry limit. If the retry limit is reached and the upload still fails, the task is marked as abnormal, the process is terminated, and a detailed error log is thrown. S502. Input real-time fragmentation status information, calculate the local persistent record of upload progress and make a loop judgment on whether all fragments are completed, and output an ordered merge voucher. Upload progress persistence and loop judgment: During the segmented upload process, the system updates the upload progress status in local storage or cache in real time (including persistent records of UploadId, list of completed segments, list of incomplete segments, list of failed segments, segment ETag, segment sequence number and file metadata information), thereby supporting the resume of upload tasks after network interruption, abnormal client exit or temporary service unavailability. The system continuously determines whether all fragments have been uploaded. If there are still fragments that have not been uploaded, it returns to the adaptive fragmentation strategy calculation stage to continue processing the remaining fragments. If all fragments have been uploaded, it collects the entity tag list and fragment sequence number of all successful fragments, and arranges them according to the fragmentation order to obtain an ordered merge certificate. S503. Based on the ordered merging certificate, construct the complete fragment upload request message body and send the complete fragment upload instruction to trigger the merging operation of each fragment data to generate the final target file object; Once all segments have been uploaded, the system obtains the ordered merge certificate. At this point, the system completes the segment upload request message body (containing UploadId, ordered segment sequence numbers, and corresponding ETag lists, etc.). Subsequently, the system sends a command to the target storage service to complete the fragment upload, triggering the server to perform a merge operation on each fragment of data; After receiving the completion instruction, the storage service client verifies the fragment integrity, fragment order, and fragment verification information. After the verification is passed, it performs the underlying data merging to generate the final target file object. S504. Based on the final target file object, calculate the unified encapsulation and adaptation of the response format, clean up system resources and record the entire chain operation status, and output standardized file upload results and the entire chain audit log. Based on the final target file object, calculate the final file access path, file size, hash value, storage location and complete metadata information returned by the storage service, as well as the adaptation and unified encapsulation of the response format returned by different cloud vendors or different storage protocols, and finally obtain a unified encapsulation and adaptation result. Subsequently, the system cleans up the temporary context, cached data, file streams, fragment status, and invalid task records generated by this upload task, releases system resources, and records the full-link audit log of this upload (including total time, routing selection strategy, target storage node, transmission rate, number of retries, exception information, and final upload result, etc.) for subsequent audit tracing, performance analysis, and routing strategy optimization. The reliability of the upload process is improved by recording results, retrying exceptions, persisting progress, and resuming interrupted uploads. The integrity and order consistency of the final file objects are guaranteed by collecting fragmented ETags, constructing completion requests, and merging them on the server side. By unifying response encapsulation, business layer callbacks, resource cleanup, and audit log recording, the differences in responses from different storage vendors are masked, reducing business layer adaptation costs and providing data support for subsequent system operation and maintenance, performance optimization, and security auditing.

[0029] Example 2: A file fragmentation upload and dynamic routing system for multi-cloud heterogeneous environments, including: Infrastructure and Service Definition Module: Constructs an abstract infrastructure through standard business interfaces, standard configuration specifications, standard basic service capabilities, and highly cohesive infrastructure; Dynamic routing strategy configuration module: Based on the abstract infrastructure, it generates a mapping table and obtains a set of candidate storage service instances; Intelligent routing matching and decision-making module: Constructs an original feature vector based on the candidate storage service instance set, uses the extreme value normalization method to obtain a standardized feature vector, and calculates the comprehensive score result of the candidate storage nodes by combining the standardized feature vector with the priority configuration of the current business scenario. Based on the comprehensive score result, it performs fault circuit breaking and automatic switching, outputs the routing result after circuit breaking, and finally performs fragmented upload initialization and concurrent upload based on the routing result after circuit breaking, outputting the target storage service instance. The process control and fragmentation strategy module reads and outputs target storage service information through the target storage service instance, performs adaptive fragmentation strategy calculation based on the file size threshold in the target storage service information, obtains the optimal fragment size, generates a fragmentation index list, performs fragmentation upload initialization and concurrent upload based on the fragmentation index list, and obtains a fragmentation upload session. The execution closing module takes the input of the fragmented upload session, first extracts the real-time fragmented status information, then verifies the collected orderly merging vouchers, then triggers the server to merge and generate the final target file object, and finally outputs the standardized file upload result and full-link audit log delivered to the business layer.

[0030] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions will not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for file fragmented upload and dynamic routing in multi-cloud heterogeneous environments, characterized in that, Includes the following steps: S1. Construct an abstract infrastructure; S2. Based on the abstract infrastructure, generate a mapping table and obtain a set of candidate storage service instances; S3. Based on the candidate storage service instance set, obtain the target storage service instance, which includes the following steps: S301. Construct the original feature vector based on the candidate storage service instance set, and use the extreme value normalization method to obtain the standardized feature vector; S302. Based on standardized feature vectors and combined with the priority configuration of the current business scenario, calculate the comprehensive score result of the candidate storage nodes; S303. Based on the comprehensive scoring results, perform fault circuit breaking and automatic switching, and output the routing results after circuit breaking; S304. Based on the routing results after the circuit breaker is triggered, perform fragmented upload initialization and concurrent upload, and finally output the target storage service instance; S4. Calculate the sharding strategy using the target storage service instance to obtain a sharded upload session. This includes the following steps: S401. Read and output target storage service information through the target storage service instance; S402. Based on the file size threshold in the target storage service information, perform adaptive sharding strategy calculation, obtain the shard size, and generate a shard index list; S403. Based on the fragment index list, perform fragment upload initialization and concurrent upload, and obtain the fragment upload session; S5. Through the fragmented upload session, calculate the fragmented status information, ordered merging vouchers, the final target file object, and the unified encapsulation adaptation result, and output the standardized file upload result and the full-link audit log.

2. The file fragmentation upload and dynamic routing method for multi-cloud heterogeneous environments according to claim 1, characterized in that, In step S1, the abstract infrastructure specifically includes: standard business interfaces, standard configuration specifications, standard basic service capabilities, and highly cohesive infrastructure.

3. The file fragmentation upload and dynamic routing method for multi-cloud heterogeneous environments according to claim 2, characterized in that, In step S1, the high-cohesion infrastructure specifically refers to: 1) Initiate business requests through the top-level cloud-native application, calculate the unified access path for the business requests, and obtain the unified access point; 2) By using a unified access point, the unified encapsulation result of request parameters, interface protocols, and access rules is calculated to obtain standardized business requests; 3) Receive standardized business requests through the load balancer configurator, calculate traffic distribution rules, task scheduling rules, and storage node matching rules, and obtain the request distribution results; 4) Based on the request distribution results, calculate the task bearing relationship between the metadata server, Hypertext Transfer Protocol node, standard protocol node, multi-cloud object storage instance and horizontally scaled object storage node to obtain the parallel heterogeneous service and storage medium layer; 5) Through parallel heterogeneous service and storage media layers, the aggregation relationship between each heterogeneous node and the underlying storage resources is calculated to obtain the object pool; 6) By using an object pool to uniformly aggregate physical storage media, cloud storage media, and multi-protocol storage nodes, the logical unity relationship of the underlying storage resources is calculated, resulting in a logically unified set of storage resources for physical and cloud storage media.

4. The file fragmentation upload and dynamic routing method for multi-cloud heterogeneous environments according to claim 1, characterized in that, In step S2, by inheriting the standard configuration specifications and standard basic service capabilities in the pre-abstract infrastructure, third-party storage services are obtained. Based on the third-party storage services and combined with preset business scenario tags, a mapping relationship table between business scenarios and third-party storage services is calculated. The uploaded resource categories are identified by comparing the set business scenario identifiers with the mapping relationship table, and finally a set of candidate storage service instances is output.

5. The file fragmentation upload and dynamic routing method for multi-cloud heterogeneous environments according to claim 1, characterized in that, The original feature vector in step S301 includes: storage cost coefficient, network latency coefficient, service reliability coefficient, and security compliance level.

6. The file fragmentation upload and dynamic routing method for multi-cloud heterogeneous environments according to claim 5, characterized in that, In step S302, the formula for calculating the comprehensive score result is as follows: ； in, This is the overall score result for the candidate storage nodes. These are the eigenvalues ​​of the normalized storage cost coefficient. These are the eigenvalues ​​of the normalized network delay coefficients. These are the characteristic values ​​of the normalized service reliability coefficient. These are the characteristic values ​​of the normalized security compliance level. The routing weights of the candidate storage nodes. These are the dynamic weighting coefficients corresponding to each evaluation indicator.

7. The file fragmentation upload and dynamic routing method for multi-cloud heterogeneous environments according to claim 1, characterized in that, In step S402, the adaptive sharding strategy calculation is specifically performed as follows: By determining whether the size of the file to be uploaded exceeds a preset segmentation threshold, it is decided whether the current file should be uploaded in normal direct upload mode or segmented upload mode: If the file size does not exceed the fragmentation threshold, the normal upload interface of the target storage service instance is called directly to upload the entire file. If the file size exceeds the fragmentation threshold, an adaptive fragmentation strategy is further calculated based on the real-time network bandwidth estimation results, and operations such as task initialization, context establishment, concurrent segmentation, data integrity verification, request encapsulation, and dynamic forwarding are performed.

8. The file fragmentation upload and dynamic routing method for multi-cloud heterogeneous environments according to claim 7, characterized in that, In step S403, the chunked upload session includes upload task ID, file metadata, total number of chunks, chunk index list, uploaded chunk status, chunk verification information, target storage node information, and chunked upload request packet.

9. The file fragmentation upload and dynamic routing method for multi-cloud heterogeneous environments according to claim 1, characterized in that, In step S5, based on the fragmented upload session, the monitoring and abnormal backoff retry of the fragmented upload results are calculated, and real-time fragmented status information is output. Input the real-time fragmentation status information. If there are still fragments that have not been completed, return to the adaptive fragmentation strategy calculation stage to continue processing the remaining fragments. If all fragments have been uploaded, collect the entity tag list and fragment sequence number of all successfully fragmented fragments, and arrange them according to the fragmentation order to obtain an ordered merge certificate. Based on the ordered merging certificate, a fragment upload request message body is constructed, and a fragment upload completion instruction is sent to trigger the merging operation of each fragment data to generate the final target file object; Based on the final target file object, calculate the final file access path, file size, hash value, storage location and complete metadata information returned by the storage service, as well as the adaptation and unified encapsulation of the response format returned by different storage protocols, and finally obtain a unified encapsulation and adaptation result.

10. A file fragmentation upload and dynamic routing system for multi-cloud heterogeneous environments, characterized in that: The system is used for the file fragment upload and dynamic routing method for multi-cloud heterogeneous environments as described in any one of claims 1-9, and the system includes: Infrastructure and Service Definition Module: Standard business interfaces, standard configuration specifications, standard basic service capabilities, and highly cohesive infrastructure to build an abstract infrastructure; Dynamic routing strategy configuration module: Based on the abstract infrastructure, it generates a mapping table and obtains candidate storage service instances; Intelligent routing matching and decision-making module: Constructs an original feature vector based on the candidate storage service instance set, uses the extreme value normalization method to obtain a standardized feature vector, and calculates the comprehensive score result of the candidate storage nodes by combining the standardized feature vector with the priority configuration of the current business scenario. Based on the comprehensive score result, it performs fault circuit breaking and automatic switching, outputs the routing result after circuit breaking, and finally performs fragmented upload initialization and concurrent upload based on the routing result after circuit breaking, outputting the target storage service instance. The process control and fragmentation strategy module reads and outputs target storage service information through the target storage service instance, performs adaptive fragmentation strategy calculation based on the file size threshold in the target storage service information, obtains the fragment size, generates a fragmentation index list, performs fragmentation upload initialization and concurrent upload based on the fragmentation index list, and obtains the fragmentation upload session. The execution closing module takes the input of the fragmented upload session, first extracts the real-time fragmented status information, then verifies the collected orderly merging vouchers, then triggers the server to merge and generate the final target file object, and finally outputs the standardized file upload result and full-link audit log delivered to the business layer.

Images