Cloud storage resource unified access and management method combining intelligent gateway and protocol conversion

Abstract

The invention belongs to the technical field of cloud computing storage, and particularly relates to a cloud storage resource unified access and management method combining an intelligent gateway and protocol conversion. The method comprises the following steps: firstly, collecting real-time characteristic parameters such as instantaneous available bandwidth and read-write concurrency of cloud storage resources, binding the real-time characteristic parameters with a native protocol identifier, and establishing a two-dimensional index characteristic protocol association library containing an updating mechanism; secondly, the intelligent gateway analyzes the service request, screens target resources through a three-dimensional matching model, and generates a customized conversion rule containing four core modules; and finally, loading the rule to complete protocol conversion, dynamically adjusting transmission parameters, monitoring protocol change in a dual-mode manner and updating the rule, and triggering rollback and alarm if verification fails. According to the method, unified access and dynamic adaptation of multi-protocol cloud storage can be realized, the request and resource adaptation accuracy and transmission stability are improved, the service continuity is guaranteed, and the management efficiency and reliability are optimized.

Description

Technical Field

[0001] This invention relates to the field of cloud computing storage technology, specifically to a unified access and management method for cloud storage resources that combines intelligent gateways and protocol conversion. Background Technology

[0002] With the rapid development of cloud computing technology, cloud storage resources, with their advantages of high scalability, flexible deployment, and on-demand use, have become the core carrier for data storage and management of various businesses. Cloud storage services provided by different vendors often use different native protocols, and the performance characteristics of cloud storage resources change dynamically with business load. This makes unified access and efficient management of multi-source cloud storage resources a key requirement for technology implementation. To achieve seamless integration of cloud storage resources with different protocols, accurate adaptation of business requests and resource performance, and stable operation and maintenance throughout the entire lifecycle, there is an urgent need for a unified access and management solution for cloud storage resources that combines intelligent gateways and protocol conversion technologies to provide efficient and reliable technical support for data storage services in multiple scenarios.

[0003] However, existing technologies fail to achieve deep binding between cloud storage resource characteristics and native protocols, and lack a unified associated data management mechanism. This results in insufficient accuracy in adapting business requests to cloud storage resources, failing to fully leverage the performance advantages of different resources. Furthermore, existing technologies often use fixed configurations for protocol conversion rules, lacking the ability to dynamically adapt to changes in the real-time status of cloud storage resources and protocol identifiers. When resource performance fluctuates or protocols are updated, issues such as decreased transmission efficiency and adaptation failures can easily occur. Moreover, there is no robust rollback and alarm mechanism to ensure business continuity.

[0004] Therefore, this invention proposes a unified access and management method for cloud storage resources that combines intelligent gateways and protocol conversion. Summary of the Invention

[0005] To address the technical problems mentioned in the background art, such as the difficulty in unified access adaptation caused by the differences in native protocols of multi-source cloud storage resources and the dynamic fluctuations in performance characteristics, as well as the fixed protocol conversion rules, untimely response to protocol changes, and lack of a sound anomaly protection mechanism, the purpose of this invention is to provide a unified access and management method for cloud storage resources that combines intelligent gateways and protocol conversion.

[0006] To achieve the above objectives, the present invention adopts the following technical solution:

[0007] A unified access and management method for cloud storage resources that combines intelligent gateways and protocol conversion includes:

[0008] S1: Collect real-time characteristic parameters of cloud storage resources, and bind the real-time characteristic parameters with the native protocol identifier of cloud storage resources to generate associated data and build a characteristic protocol association library;

[0009] S2: The smart gateway receives and parses the business request, extracts the operation type and data volume, filters the target cloud storage resources based on the characteristic protocol association library, and after extracting the associated data, triggers the engine to generate customized conversion rules.

[0010] S3: Load the customized conversion rules to complete the protocol conversion, collect resource load and transmission performance parameters in real time to update the feature protocol association library, dynamically adjust the conversion rate and data format synchronously, and apply the priority strategy until the data transmission is completed;

[0011] S4: Monitor changes in the target resource protocol identifier, collect new protocol features and real-time characteristic parameters, generate new related data and rules, synchronize and update to the characteristic protocol association library and verify them. If it fails, it will trigger rollback and alarm.

[0012] Furthermore, in step S1, the real-time characteristic parameters include instantaneous available bandwidth, read / write concurrency, and storage medium type. These parameters are collected through standardized detection commands. Abnormal data is removed during the collection process, and the effective values ​​are calculated after noise reduction. The characteristic protocol association library adopts a two-dimensional index structure that combines the native protocol identifier with the device's unique identifier to establish a query index for key characteristic parameter fields and sets a data validity period and a periodic scanning and update mechanism.

[0013] Furthermore, the storage medium type is coded and classified according to preset rules, the native protocol identifier covers mainstream cloud storage protocol types, the real-time characteristic parameters are bound to the native protocol identifier to form a structured data group, and each data group is accompanied by a timestamp; the detection command is sent through a smart gateway transmission interface compatible with general network protocols, and the resource identifier, network address and command sending time information are recorded during the sending process. Resources that do not respond within the timeout period will trigger a retry mechanism or be marked as abnormal resources.

[0014] Furthermore, in step S2, after receiving the service request, identity authentication and format verification are completed first. For legitimate requests, a unique identifier and the network information of the initiating party are extracted. The operation type is classified according to the data processing behavior, and the data volume is divided into multiple levels according to the total data volume. The total volume of multiple data sets is calculated by accumulating the volume of a single data set. The target resource is screened through a three-dimensional matching model of operation type, data volume, and resource characteristics. The final target resource is determined by combining storage medium compatibility, performance compatibility coefficient, transmission time redundancy, and bandwidth redundancy.

[0015] Furthermore, the performance adaptation coefficient calculation adopts a dynamic weight allocation mechanism, with the weight coefficient adjusted according to the operation type, and recalibrated based on optimization theory when the transmission performance is substandard; the customized conversion rules include four core modules: protocol mapping relationship, conversion rate threshold, data fragmentation strategy, and format adaptation standard. The protocol mapping establishes a field-level correspondence, automatically supplementing or triggering retries when necessary fields are missing. The conversion rate threshold is calculated in conjunction with the target resource bandwidth carrying capacity. The data fragmentation strategy adapts to the data volume and resource concurrent processing capability, and the format adaptation standard matches the storage medium type.

[0016] Furthermore, in step S3, before the protocol conversion is executed, the protocol parsing library is loaded, the rule syntax is checked, and the compatibility is verified. The cache priority is calculated according to the completeness, rationality, and compliance of the core rule module, and high-priority rules are cached in memory first. During the data transmission, multi-dimensional real-time parameters are collected at a fixed frequency, and after noise reduction processing, they are synchronously updated to the feature protocol association library. Based on the updated data, the conversion rate and data format are dynamically adjusted, and the adjusted rules are applied immediately or delayed according to their effective priority.

[0017] Furthermore, step S4 employs a dual-mode monitoring approach of timed detection and event listening to detect protocol changes. It calculates the protocol consistency coefficient by matching the core fields, and determines a protocol change if the coefficient is lower than a preset threshold. It collects four types of interactive features of the changed protocol: version information, field adjustments, authentication methods, and data format requirements. It then re-collects real-time characteristic parameters to generate a new structured data set. It uses the rule generation logic to trigger the engine to generate customized conversion rules adapted to the new protocol. It calculates the update priority based on the impact of field changes, the importance of authentication adjustments, and transmission stability requirements to determine when the rules take effect.

[0018] Furthermore, in step S4, the newly generated associated data and rules are synchronously updated to the feature protocol association library and assigned a unique identifier. The validity is verified by the rule execution error rate. If the error rate exceeds the threshold, a rollback mechanism is triggered to restore the old data and rules from the historical version partition of the association library. At the same time, a graded alarm is issued according to the error rate level. After the data transmission is completed, the transmission quality evaluation value is calculated. The evaluation result is stored in association with the transmission statistics and rule version, taking into account the transmission success rate, rate compliance, and rule adjustment frequency index.

[0019] Compared with the prior art, the advantages of the present invention are as follows:

[0020] 1. This invention binds the real-time characteristic parameters of cloud storage resources with native protocol identifiers to construct a characteristic protocol association library with a two-dimensional index structure, and sets data validity period and periodic update mechanism to achieve deep association management of resource characteristics and protocol information. Based on this association library, target resources are filtered by a three-dimensional matching model of operation type, data volume and resource characteristics, which greatly improves the accuracy of matching business requests with cloud storage resources and gives full play to the performance advantages of different resources.

[0021] 2. The customized conversion rules of this invention can dynamically adjust the conversion rate and data format based on the real-time status of the target resources. At the same time, it adopts a dual-mode monitoring of protocol changes by timed detection and event listening. After detecting a protocol change, it automatically collects the new protocol characteristics and resource parameters, generates new rules that are adapted and updates them synchronously. With the addition of rule execution error rate verification, historical version rollback and hierarchical alarm mechanism, it effectively solves the problems of poor adaptability of traditional fixed rules and untimely response to protocol changes, and ensures the stability and continuity of business transmission.

[0022] 3. This invention introduces a rule caching priority strategy and an effective priority mechanism during the protocol conversion process. High-priority rules are cached in memory first, key parameter adjustments take effect immediately, and non-critical adjustments are applied with a delay, balancing transmission efficiency and stability. At the same time, through multi-dimensional parameter collection, noise reduction processing, and transmission quality evaluation, a closed loop of performance monitoring and optimization is formed throughout the process, further improving the overall efficiency and reliability of unified access and management of cloud storage resources. Attached Figure Description

[0023] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1 This is a schematic diagram of the workflow of the method of the present invention;

[0025] Figure 2 This is a schematic diagram of the target resource matching and customized conversion rule generation process of the present invention;

[0026] Figure 3 This is a schematic diagram of the closed-loop process for protocol change monitoring and rule update of the present invention. Detailed Implementation

[0027] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0028] To achieve the above objectives, this invention provides a unified access and management method for cloud storage resources that combines a smart gateway with protocol conversion, such as... Figures 1-3 As shown, the method includes:

[0029] S1: Collect real-time characteristic parameters of cloud storage resources, bind the real-time characteristic parameters with the native protocol identifier of cloud storage resources to generate associated data, and build a characteristic protocol association library.

[0030] S101: After the system starts, the feature protocol association unit is activated first, and the communication interface compatibility verification, data acquisition module initialization and command transmission channel connectivity test are completed in sequence to ensure that the feature protocol association unit can stably connect to various cloud storage resources to be accessed.

[0031] Subsequently, standardized detection commands are generated based on the cloud storage resource access technical specifications. These standardized detection commands include a parameter collection list, a transmission timeout threshold, and a sampling frequency. The parameter collection list covers instantaneous available bandwidth, read / write concurrency, and storage media type. The transmission timeout threshold is set as follows: (Unit: ms), the sampling frequency is set to (Unit: times / second), and a unique instruction traceability is embedded in the detection command. ;

[0032] The standardized detection command specifies that the single-round collection time for subsequent parameter acquisition is 10 seconds. The bandwidth and concurrency will be calculated based on this time. The standardized detection command is issued through the general transmission interface of the smart gateway.

[0033] In this embodiment, based on industry-standard parameters, 5000ms meets the basic requirements for timeout tolerance in inter-device communication, and is particularly suitable for scenarios involving cross-network response latency in cloud storage resources. Therefore, the transmission timeout threshold... The value is 5000ms; according to According to the standard, to accurately reproduce signal characteristics, the sampling frequency should be at least twice the highest frequency component of the signal. In industry practice, to further improve data accuracy, a "sampling frequency = 5 times the highest frequency component" configuration strategy is often adopted. Therefore, the sampling frequency mentioned... The value is 5 times / second;

[0034] S102: Via the smart gateway's universal transmission interface (compatible) , The protocol sends standardized detection commands to the cloud storage resources to be accessed, and simultaneously records the unique device identifier of each cloud storage resource. Network access address port ( +port) and command issuance timestamp; wherein, the device's unique identifier middle , The total number of cloud storage resources to be accessed; the smart gateway plays the role of protocol conversion and data transfer in this process, supports bidirectional adaptation between external request protocols and cloud storage native protocols, and ensures the format compatibility of the detection commands and the collected data;

[0035] Pending After responding to the detection command, each cloud storage resource initiates a parameter acquisition process: within a 10-second acquisition window, the data volume of each sample is recorded at a frequency of 5 times per second. (Unit: Byte) and transmission time (Unit: s), excluding the aforementioned data volume Exceeding and the transmission time Exceeding After the data is within the range, calculate using the formula. Calculate the single bandwidth (Unit: Byte / s), then take the moving average of 10 samples, and calculate using the formula. Obtain the instantaneous available bandwidth of the resource (Unit: Byte / s);

[0036] In this embodiment, the technical specification for outlier range filtering is based on the general requirements for cloud resource monitoring in information technology cloud computing. The upper limit of the interval covers the reasonable data volume range for a single sampling by mainstream cloud storage such as Alibaba Cloud, avoiding interference from invalid empty packets and excessive abnormal data. Therefore, the data volume mentioned... The threshold interval is defined as ;

[0037] Next, read / write concurrency statistics are performed: the resource status monitoring interface is triggered to capture the number of read requests within 10 seconds. Number of write requests Duplicate requests are eliminated by comparing timestamps, and then by calculating... Calculate the resource read / write concurrency (Unit: times) );

[0038] Finally, storage medium encoding is performed: resource medium attributes are obtained through protocol interaction and encoded according to rules. , indicating the first The storage medium code of the cloud storage resource to be accessed, with a value range of [value range missing]. If the attribute is The code is then 1. The encoding is 2, and the distributed storage encoding is 3;

[0039] If the transmission timeout threshold is exceeded after the detection command is issued. If no response is received from the resource, the system will automatically retry twice, with a 1000ms interval between each retry. If no response is received after the retry, the resource will be marked as an abnormal resource to be accessed and recorded in the abnormal log.

[0040] S103: Extract the native protocol identifiers of each cloud storage resource. (include protocol, Agreement and (Agreement), and will the first Native protocol identifier of individual cloud storage resources Instantaneous available bandwidth of resources resource read / write concurrency and storage medium encoding Binding is performed to construct a four-dimensional structured data set. ,Right now And add a millisecond-level collection timestamp to each set of data. ;

[0041] S104: Based on the aforementioned four-dimensional structured data set Building a feature protocol association library: A two-dimensional index table is used as the basic framework of the library, with the first-level index set as the native protocol identifier. The secondary index is set as a unique identifier for the device. ; at the same time as the first Each cloud storage resource corresponds to Create separate query indexes for each field to support multi-condition combined retrieval;

[0042] The data validity period is set to 30 minutes. The system initiates a data scan of the database every 5 minutes, marking data that has been stored for more than 30 minutes and has not been updated as invalid data. When the next detection process starts, the system will automatically re-collect data and replace invalid items, ultimately forming a feature protocol association library. , represented as .

[0043] S2: The smart gateway receives and parses the business request, extracts the operation type and data volume, filters the target cloud storage resources based on the characteristic protocol association library, extracts the associated data, and triggers the engine to generate customized conversion rules.

[0044] S201: The smart gateway supports standardized access ports (supports...) , Upon receiving an external data service request, the protocol first initiates a request validity verification process: verifying the authentication information in the request header (such as...). The system checks whether the key (access token) matches the system's preset authorization list. The authorization list is stored encrypted, and the key is forcibly rotated every 90 days. Encryption key management specifications; and verification of request format. Standard data exchange specifications are used to reject invalid requests (such as authentication failures or incorrect formats) and record them in the request log; for valid requests, their unique identifiers are extracted. (Generated from timestamp + random string), and associated with the network address of the request initiator. With port number This is used for subsequent data transmission tracking and anomaly backtracking;

[0045] S202: By parsing the request body This field determines the type of operation requested by the external data service. The range of values ​​is The corresponding definitions are as follows: 1 = data write operation (such as file upload, data storage), 2 = data read operation (such as file download, data query), 3 = data delete operation (such as file deletion, record cleanup); at the same time, the additional parameters associated with the operation are extracted, including the storage path of the write operation, the fragment size of the read operation and the batch identifier of the delete operation, as an auxiliary basis for resource matching;

[0046] For write requests, directly read the request body. The field's numerical value (unit: Byte) is used as the original data volume. For read / delete requests, the target data identifier (such as a file) specified in the request is parsed. (Data index) Query the system's pre-stored metadata information to obtain the storage level of the corresponding data. (Unit: Byte); If the request involves multiple data items (such as batch upload / download), the total data volume is calculated using the following formula. :

[0047]

[0048] in, For the first The original data volume of the data (unit: Byte); The number of data copies involved in the request, and ;

[0049] To adapt to the performance characteristics of different cloud storage resources, the total data volume is... The data volume is divided into three levels according to the following rules. The classification results are used for subsequent resource matching:

[0050] like Byte (i.e., ≤100MB), then (Small-scale data);

[0051] like (i.e., 100MB~10GB) (Medium-level data);

[0052] like Byte (i.e., >10GB), then (Large amounts of data);

[0053] This classification references the "Data Transmission Scale and Storage Media Adaptation Specification" in GB / T37737-2019 Information Technology Cloud Computing Distributed Block Storage System General Technical Requirements, and matches the distributed storage system's optimization logic for processing data of different scales.

[0054] S203: Call the aforementioned feature protocol association library The target cloud storage resources are selected based on a three-dimensional matching model of "operation type-data volume-resource characteristics". The specific steps are as follows:

[0055] Initial filtering: based on operation type Filter the appropriate storage media type Write / read operations are matched first. (SSD) or (Distributed storage), the deletion operation has no special restrictions on the storage medium. (Both are acceptable); at the same time, cloud storage resources corresponding to data marked as invalid in the feature protocol association library are removed to ensure the validity of the matching objects;

[0056] Performance adaptation calculation: For the initial set of candidate cloud storage resources ( (where the number of candidate resources is 1), calculate the performance fit coefficient for each candidate resource. The formula is as follows:

[0057]

[0058] in, For the first Instantaneous available bandwidth of each candidate resource (unit: Byte / s, taken from the feature protocol association library) Data set); For the first Real-time load rate of each candidate resource; This represents the maximum instantaneous available bandwidth (in bytes / s) in the candidate resource set. For the first Read / write concurrency of candidate resources (unit: times / s, taken from the feature protocol association library) Data set); This represents the maximum read / write concurrency in the candidate resource set (unit: times / s). Adjust weights to fit the data volume. Bandwidth-adapted weighting coefficients, Weighting coefficients for read / write concurrency To adapt the weighting coefficients to the data volume, and satisfy ;

[0059] The weighting coefficient , , The conditions for triggering the adjustment are as follows: when the system detects that the success rate of three consecutive requests of the same operation type is lower than 99.5%, or the average transmission delay exceeds the preset threshold (500ms for write / read operations, 1s for delete operations), the coefficient values ​​are recalibrated based on Pareto optimality theory to ensure a balance between efficiency and fairness in resource matching.

[0060] In this embodiment, the real-time load rate Range of values Taken from the feature protocol association library, for The average of occupancy rate and storage utilization rate;

[0061] Based on the load adaptation theory and industry practice standards of cloud storage resource scheduling, the data volume adaptation weight is... Based on data volume The value is determined: hour , hour , hour Weighting coefficients , , The algorithm dynamically adjusts based on the operation type, prioritizing bandwidth and concurrency during write / read operations. , , This allocation method matches the typical weighting strategy of cloud storage I / O resource scheduling. Bandwidth and concurrency are the core indicators affecting read and write operation performance. Equal weighting of the two can balance transmission efficiency and concurrency carrying capacity. When deleting, priority is given to data volume adaptation. , , This weighting allocation aligns with optimization practices for large-scale data deletion. The data volume directly determines the resource consumption and execution risk of the deletion operation, so a higher weight is assigned, while a small amount of bandwidth and concurrency weight is reserved to avoid operation blocking.

[0062] Target resource determination: Candidate resources are determined according to the aforementioned performance adaptation coefficient. Sort in descending order and select the performance fit coefficient. The two largest resources are selected as candidate resources; if there is only one candidate resource, it is directly selected as the target resource.

[0063] For the two candidate resources, first calculate the theoretical minimum transmission time. ,like (Reserving 20% ​​time redundancy), directly exclude this candidate resource; then calculate the matching redundancy. Select (i.e., bandwidth redundancy ≥ 20%) of the alternative resources will be used as the final target cloud storage resources; among which... The maximum acceptable transmission time preset for external requests (unit: seconds, taken from the Max-Trans-Time field in the request header); if both alternative resources meet the redundancy requirements, then the performance adaptation coefficient is selected. Larger resources; if none of these conditions are met, the selection range is expanded again from the feature protocol association library (e.g., relaxing storage medium restrictions) until the target resource is determined;

[0064] S204: From the aforementioned feature protocol association library Extract the complete characteristic protocol association data corresponding to the target cloud storage resource, whose structure is a four-dimensional structured data set. :

[0065]

[0066] in, The native protocol identifier of the target cloud storage resource (such as S3, Swift, OSS protocol); The instantaneous available bandwidth of the target cloud storage resource (unit: Byte / s); The read / write concurrency of the target cloud storage resource (unit: times / s); The storage medium encoding for the target cloud storage resource (1=SSD, 2=HDD, 3=distributed storage);

[0067] Simultaneously extract the four-dimensional structured data group. Collection timestamp Used to determine the timeliness of data: if the current time is... The difference exceeds This triggers the parameter acquisition process in step S1 to update the four-dimensional structured data set. ;in, Dynamically set based on operation type: write / read operation =5 minutes (for high time sensitivity), delete operation =10 minutes (for low timeliness requirements), refer to the best practice cycle for distributed storage node status scanning;

[0068] S205: Based on the aforementioned four-dimensional structured data set Based on the request parsing results, the protocol conversion strategy engine is triggered to generate customized conversion rules adapted to the target cloud storage resources. The rules include four core modules: protocol mapping relationship, conversion rate threshold, data sharding strategy, and format adaptation standard.

[0069] The protocol mapping relationship is based on the native protocol identifier. Protocol with external request source (Taken from the Source-Protocol field in the request header), establish a field-level mapping table to clarify the correspondence between source protocol fields and target protocol fields (such as the adaptation of request header fields, data body structure, and verification rules). If a source protocol field is missing a required field of the target protocol, the supplementary triggering condition will be automatically triggered. This will be adjusted to: "If a source protocol field is missing a required field of the target protocol, and the field supports default value configuration (such as storage region, storage type), then the default parameters of the target cloud storage will be automatically filled in; if the field is not a default required field (such as authentication information, data encryption identifier), then the request retry mechanism will be triggered, with a maximum of 2 retries, each with an interval of 500ms. If the retry fails, a field missing error message will be returned."

[0070] This embodiment provides an example: if , Then The request's Content-Length field is mapped to the S3 protocol's x-amz-content-length field, while also supplementing the signature authentication field required by the S3 protocol;

[0071] The conversion rate threshold is calculated by combining the instantaneous available bandwidth of the target cloud storage resource. With the total data volume The maximum rate threshold for protocol conversion is calculated using the following formula. (Unit: Byte / s) Ensure that the conversion rate does not exceed the bandwidth capacity of the target resource:

[0072]

[0073] Among them, 0.9 is the rate safety factor, reserving 10% bandwidth to cope with bandwidth fluctuations during transmission; This represents the performance overhead coefficient for protocol conversion.

[0074] In this embodiment, based on the industry-standard strategy of reserving network transmission bandwidth, reserving 10% bandwidth redundancy is a typical solution for balancing transmission efficiency and stability. This avoids the bandwidth being completely saturated, which could lead to uncontrollable bursts of traffic, while ensuring that the main bandwidth resources are used for data transmission. It also conforms to the common configuration recommendation that the transmission rate should not exceed 90% of the available resource limit. Therefore, the rate safety factor is set to 0.9. Based on measured data of resource consumption from protocol conversion, analysis of cloud storage protocol interaction complexity, and industry optimization practices, the performance overhead factor... Range of values The complexity is determined based on the conversion complexity between the source and target protocols: Converting to S3 takes 0.15, and conversion between simple protocols takes 0.1.

[0075] Simultaneously set the minimum rate threshold Byte / s (i.e. 1MB / s). When the actual conversion rate is lower than this value, the smart gateway's resource scheduling optimization is triggered (such as releasing redundant processes and prioritizing CPU resources).

[0076] The data sharding strategy is based on the data volume level. Read / write concurrency of the target cloud storage resource The number of fragments can be calculated using the following formula. Compared to single fragment size (Unit: Byte), the calculation formula is:

[0077]

[0078]

[0079] in, The maximum single fragment size supported by the target protocol (unit: Byte); The minimum single fragment size supported by the target protocol (in bytes); This represents the floor function. This represents the floor function; Set a maximum number of shards (reserve 50% redundancy based on the target resource concurrency to avoid overload of concurrent requests);

[0080] In this embodiment, based on the sharding specifications of mainstream cloud storage protocols, the maximum single shard size is... Depend on Sure: The agreement is Byte, The protocol is 5GB. The protocol is 10GB = 10 × 10^{9} bytes; the minimum single fragment size supported by the target protocol. Depend on Confirmed: S3 protocol is 5MB Byte, Swift protocol is 5MB, OSS protocol is 1MB Byte);

[0081] This embodiment provides an example:

[0082] like Byte (8GB), ( Byte), times / s, then , Byte (4GB);

[0083] Format compatibility standards: According to (Storage media type) Determines the data compression and verification format: If (SSD), using Compression algorithm (balancing compression efficiency and decompression speed), verification method is: ;like (HDD), using Compression algorithm (higher compression ratio), verification method is MD5; if (Distributed storage), using Compression algorithm, verification method is At the same time, specify the encoding format for data transmission (e.g., Adaptation requirements such as byte order (big-endian / little-endian) are met to ensure that data can be read and parsed normally after being written to the target resource;

[0084] The final generated customized conversion rules are based on Formatted storage, with a unique identifier for the rule. (Request unique identifier + target protocol identifier), used for subsequent calls to the protocol conversion execution module and rule management.

[0085] S3: Load the customized conversion rules to complete the protocol conversion, collect resource load and transmission performance parameters in real time to update the feature protocol association library, dynamically adjust the conversion rate and data format synchronously, and apply the priority strategy until the data transmission is completed.

[0086] S301: The core control unit of the smart gateway sends a trigger command to the protocol conversion execution module, carrying the customized conversion rules generated in step S2 (including field mapping table, rate threshold, fragmentation configuration and protocol adaptation parameters) and the target cloud storage resource identifier. At the same time, the module loads the native protocol parsing library of the corresponding target cloud storage (such as S3 protocol parsing library and OSS protocol adaptation plugin). After completing the environment initialization, the module performs syntax verification and compatibility verification on the customized conversion rules.

[0087] After successful verification, calculate the rule cache priority. To ensure that frequently called parameters are loaded first, the calculation formula is:

[0088]

[0089] in, Map weight coefficients to fields; This is the rate threshold weighting coefficient; Configure weight coefficients for shards; For field mapping integrity; To ensure the reasonableness of the rate threshold; Configure compliance for sharding;

[0090] Rules with a speed lower than 0.8 are cached directly in local memory, while rules with a speed lower than 0.8 are temporarily stored on disk to improve the calling efficiency of high-priority rules.

[0091] In this embodiment, based on the "GB / T37738-2019 Technical Requirements for Distributed Caching Systems", the rule-based cache priority is... The range of values ​​is The larger the value, the higher the priority; based on the "Cloud Storage Protocol Conversion Middleware Performance White Paper", the field maps to a weight coefficient. The value is 0.4, which is the rate threshold weighting coefficient. and the partition configuration weight coefficient All values ​​are 0.3; based on the "Field Matching Algorithm for Protocol Conversion", the field mapping integrity is... The range of values ​​is The value is 1 when the field matching rate is 100%; based on "YD / T3868-2021 Cloud Computing Data Center Network Bandwidth Test Method", the rationality of the stated rate threshold is questionable. The range of values ​​is The value is 1 within the range of [1MB / s, 100MB / s]; the sharding configuration compliance. The range of values ​​is 1 is taken when the fragmentation specification conforms to the target protocol.

[0092] S302: Extract the source protocol type of external data service requests ( ) and data The conversion is performed based on the field mapping table, specifically as follows:

[0093] Complex protocol conversion (such as) (Transfer to S3): Automatically supplements the required fields of the target protocol (such as x-amz-date in S3), generates signature authentication information, and calculates the data integrity check value. The calculation formula is:

[0094]

[0095] in, The hash value is used to verify data integrity. This is the original data payload; A unique random salt value specific to the target cloud storage, taken from the feature protocol association library;

[0096] After conversion, the hash value is verified through the data integrity check. The data transmission integrity is verified by comparing the calculation results with those of the target cloud storage receiver; it supports multi-format data encoding conversion, and after completing the conversion according to the target format specified by the rules, it performs format verification. After the verification is passed, a conversion completion receipt is generated, and the data transmission is ready to be started.

[0097] S303: After data transmission is started, the multi-dimensional parameter acquisition module is activated, and real-time characteristic parameters of the target cloud storage resources are collected at a sampling frequency of 500ms / time. The collection scope includes resource load, transmission performance and protocol interaction.

[0098] The resource load class includes Utilization rate, memory usage rate, storage nodes Business and current concurrent connections; the transmission performance category includes actual bandwidth utilization, data transmission rate, packet retransmission rate, and transmission latency; the protocol interaction category includes protocol response time, field parsing success rate, and authentication success efficiency;

[0099] The original parameters are denoised to remove outliers; the calculation formula is as follows:

[0100]

[0101] in, These are the effective values ​​of the parameters after noise reduction; The number of samples, and ; For the first The original sampled values ​​are then used to remove the first and last abnormal samples.

[0102] The effective value of the noise-reduced parameters Synchronize the data to the feature protocol association library in the format of "resource identifier-parameter type-collection timestamp-parameter value" as the core basis for subsequent rule adjustments;

[0103] S304: Based on real-time transmission parameters, the adjusted target rate is calculated using a formula:

[0104]

[0105] in, To adjust the target conversion rate; This represents the current actual transmission rate. This is a weighting coefficient for bandwidth utilization. This is the weighting coefficient for the data packet retransmission rate; This represents the current bandwidth utilization rate. This is the bandwidth utilization threshold; This represents the current packet retransmission rate. This is the retransmission rate threshold;

[0106] The adjusted target conversion rate The constraints are ;in, Byte / s (1MB / s), The maximum rate threshold for protocol conversion described in step S2. ;

[0107] when or At that time, according to the adjusted target conversion rate Correct the conversion rate and simultaneously coordinate with the resource scheduling module to allocate corresponding resources. / Memory resources should be used to ensure that the speed remains stable within a reasonable range;

[0108] In this embodiment, based on industry practices of cloud storage transmission scheduling, the bandwidth utilization weighting coefficient... The value is 0.6; based on the cloud computing network resource scheduling specification, the data packet retransmission rate weighting coefficient... The value is 0.4; based on the data transmission reliability test standard, the bandwidth utilization threshold is... The value is set to 0.8; the retransmission rate threshold. The value is 0.05;

[0109] S305: Determine whether to switch data formats by calculating the storage node load adaptability coefficient.

[0110]

[0111] in, This refers to the load adaptability factor. For storage nodes Busyness level; This represents the original size of a single data packet. This represents the data compression ratio, and the ratio of the size of the lightweight format to the original format.

[0112] when When switching to a lightweight format (such as...) );when When, restore the original specified format; when At this time, maintain the current format;

[0113] Based on the load adaptability coefficient The calculation results can be smoothly switched to different data formats, reducing the parsing pressure on storage nodes while ensuring data readability.

[0114] S306: The adjusted rules calculate the priority of application using the following formula to ensure that key parameters are applied first:

[0115]

[0116] in, Priority for rule implementation; The formula for calculating the rate adjustment range is: ; The formula for calculating the variation range of the adaptation coefficient is as follows: ;

[0117] If the rule takes effect, priority The rules take effect immediately; if the rules take effect with priority... The rules are delayed until the next data fragment is transmitted to avoid frequent adjustments affecting transmission stability;

[0118] S307: When a "transmission confirmation" response is detected from the target cloud storage, calculate the transmission quality assessment value:

[0119]

[0120] in, This is a transmission quality assessment value; The transmission success rate is set to 1 for successful transmission and 0 for failure. This represents the average transmission rate. The expected rate is the maximum rate threshold for protocol conversion described in step S2. 80%; Adjust the number of times the rule is set; The total number of data shards;

[0121] Stop the parameter acquisition and rule adjustment process, and change the transmission quality assessment value. Transmit statistical information and the final rule version, associate them and store them in the system log library, and return the response results to the external business requester;

[0122] In this embodiment, based on the cloud computing data transmission quality assessment specification, the transmission quality assessment value is... The range of values ​​is .

[0123] S4: Monitor changes in the target resource protocol identifier, collect new protocol features and real-time characteristic parameters, generate new related data and rules, synchronize and update to the characteristic protocol association library and verify them. If it fails, it will trigger rollback and alarm.

[0124] S401: After the smart gateway starts, the protocol change monitoring unit automatically initializes, loads the native protocol monitoring plugins for the target cloud storage resources (such as the S3 protocol change monitoring plugin and the OSS protocol version adaptation module), and completes the monitoring environment configuration.

[0125] The unit reads the target cloud storage resource identifier and the native protocol identifier determined in step S2. Monitoring is initiated through a dual-mode approach combining timed detection and event listening.

[0126] The timed detection sends a protocol version detection command to the target cloud storage resource at a frequency of 30 seconds / time. The command includes a native protocol identifier verification field to obtain the current protocol version information. The event listening receives protocol change events (such as the addition of protocol fields or the adjustment of authentication methods) actively pushed by the target cloud storage by registering a cloud storage resource protocol change notification interface. The monitoring unit performs format verification on the data returned by the detection command and the notification interface to ensure data integrity and avoid abnormal data from interfering with the change judgment.

[0127] S402: Calculate the protocol consistency coefficient to determine if a change has occurred.

[0128]

[0129] in, This is the protocol consistency coefficient (value from 0 to 1); The total number of core fields of the target protocol (e.g., the S3 protocol has 20 core fields, including authentication fields and data interaction fields); The first one detected so far One core field; The number recorded in step S2 One core field; This is a field matching function; it returns 1 if the fields match exactly, and 0 if they do not match.

[0130] If the protocol consistency coefficient When the core field matching rate is below 90%, it is determined that the protocol has changed; the protocol consistency coefficient If the agreement remains unchanged, monitoring will continue.

[0131] If a protocol change is detected, the monitoring unit automatically triggers the feature collection process, sends an interaction feature detection packet to the target cloud storage resource, and collects the core information of the changed protocol: new protocol version number, field addition / deletion / modification information, authentication method adjustment content, data format requirements, and interaction timeout threshold.

[0132] The collected interaction features are deduplicated and categorized, and a structured table of protocol interaction features is generated in the format of "field name-field type-mandatory-adaptation requirements" to ensure full coverage of the interaction specifications of the changed protocol.

[0133] S403: Invoke the parameter acquisition logic of step S1 to re-acquire the real-time characteristic parameters of the target cloud storage resource, including instantaneous available bandwidth, read / write concurrency, and storage media type, ensuring the compatibility of the parameters with the changed protocol. Combine the interaction characteristics of the changed protocol with the re-acquired real-time characteristic parameters to generate a new four-dimensional structured data set. The expression is:

[0134]

[0135] in, The native protocol identifier for the target cloud storage resource after the change; The instantaneous available bandwidth (in bytes / s) for the target resource to be re-collected; Concurrency of read / write operations for the target resource to be re-collected (unit: times / s); Encode the storage medium for the target resource;

[0136] For the new four-dimensional structured data group Add millisecond-level collection timestamps Mark the data generation time for subsequent timeliness assessment;

[0137] S404: Based on the new four-dimensional structured data set The interaction characteristics with the modified protocol trigger the protocol conversion strategy engine to generate customized conversion rules adapted to the new protocol. The core rules module has been adjusted as follows:

[0138] The protocol mapping relationship is updated based on the field requirements of the new protocol, updating the field-level mapping table, adding new fields, deleting obsolete fields, and adjusting the adaptation rules for modified fields; the conversion rate threshold is calculated according to the rate calculation logic of step S2, combined with the instantaneous available bandwidth of the target resource. The maximum conversion rate threshold is recalculated using the following formula:

[0139]

[0140] in, This is the performance overhead coefficient for the protocol conversion after the change (adjusted according to the complexity of the new protocol, with a value of 0.1~0.2);

[0141] The data fragmentation strategy is as follows: if the new protocol has new requirements for fragment size, update the maximum fragment size of the new protocol. With minimum fragment size The number of fragments and the size of a single fragment are recalculated. The new protocol fragment size requirement is taken from the protocol interaction feature structured table generated in S402.

[0142] If the new protocol has new provisions on data compression and verification formats, the compression algorithm and verification method shall be adjusted according to the new requirements.

[0143] right Perform syntax validation and actual interaction tests to simulate external data service requests, verifying whether the rules can achieve normal conversion and data transmission of the new protocol, and ensuring no interaction errors; the rule update priority is determined by the following formula:

[0144]

[0145] in, Prioritize rule updates; This refers to the weighting coefficient for the impact of field changes. Adjust the importance weighting coefficient for the authentication method; Weighting coefficients for transmission stability requirements; Impact of field changes; Adjust the importance of the authentication method; For transmission stability requirements;

[0146] In this embodiment, based on cloud storage protocol adaptation practices, the impact weighting coefficient of the field change is... The value is 0.4, and the importance weight coefficient of the authentication method is adjusted. The value is 0.3, which is the weighting coefficient for transmission stability requirements. The value is 0.3; the impact of the field change. The range of values ​​is Changes to core fields are assigned a value of 1, while changes to non-core fields are assigned a value of 0.3; the importance of the authentication method is adjusted. The value is 0 or 1; 1 indicates a change in authentication method, and 0 indicates no change; the transmission stability requirement... Range of values Based on the target resource read / write concurrency Calculation, concurrent operations ≥ 100 times Take 1, otherwise take 0.5;

[0147] When the rule updates priority When the new rule is implemented, it immediately overrides the old rule. This will take effect after the current fragment transmission is completed; When needed, the update will be delayed until off-peak hours (concurrency < 500 times / s) to balance the timeliness of the update with business continuity;

[0148] The new four-dimensional structured data group , and the rule update priority Synchronize and update to the feature protocol associated library Replace the original and ;

[0149] for Assign a new rule unique identifier ;in, To request a unique identifier; A new protocol identifier is used to ensure the uniqueness of rule invocations;

[0150] The system sends a rule update notification to the core control unit of the smart gateway, informing it of when the new rule will take effect. Subsequent data transmission will follow the update priority of the rule. The corresponding strategy loads the unique identifier of the rule. implement;

[0151] S405: After receiving the update notification, the core control unit of the smart gateway sends a rule loading instruction to the protocol conversion execution module to verify the unique identifier of the rule. The loading and execution status is monitored. If loading is successful, an update confirmation receipt is returned, completing the update loop.

[0152] The anomaly detection formula is used to determine the anomaly. The formula is as follows: ;in, The rule execution error rate (%) per unit time; This represents the number of rule execution errors per unit of time. This represents the total number of times the rule is executed per unit of time.

[0153] when When this occurs, it is determined that the new rule verification has failed (such as protocol conversion error or data transmission interruption), which meets the standard alarm threshold for cloud resource monitoring.

[0154] The rollback mechanism is triggered, and the system automatically reverts to the state before the update. and To ensure business continuity, before the update and Historical versions of the feature protocol associated library have been backed up to this partition and will be read from this partition during rollback; exception information (including protocol change content, rule verification error logs, etc.) will be included. Calculation results The numerical values ​​are synchronized to the system's exception database to create a traceable exception file;

[0155] Trigger a tiered alarm, press The alarm levels are determined by numerical values ​​(5%~10% is a Level 1 alarm, >10% is a Level 2 alarm), and technical personnel are notified to prioritize the investigation of higher-level issues.

[0156] After the rollback, the protocol change monitoring unit will continue monitoring at the original monitoring frequency until the new rules are optimized, verified, and updated.

[0157] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0158] In conclusion, the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A unified access and management method for cloud storage resources combining intelligent gateways and protocol conversion, characterized in that: include: S1: Collect real-time characteristic parameters of cloud storage resources, and bind the real-time characteristic parameters with the native protocol identifier of cloud storage resources to generate associated data and build a characteristic protocol association library; S2: The smart gateway receives and parses the business request, extracts the operation type and data volume, filters the target cloud storage resources based on the characteristic protocol association library, and after extracting the associated data, triggers the engine to generate customized conversion rules. S3: Load the customized conversion rules to complete the protocol conversion, collect resource load and transmission performance parameters in real time to update the feature protocol association library, dynamically adjust the conversion rate and data format synchronously, and apply the priority strategy until the data transmission is completed; S4: Monitor changes in the target resource protocol identifier, collect new protocol features and real-time characteristic parameters, generate new related data and rules, synchronize and update to the characteristic protocol association library and verify them. If it fails, it will trigger rollback and alarm.

2. The method for unified access and management of cloud storage resources combining intelligent gateway and protocol conversion according to claim 1, characterized in that, In step S1, the real-time characteristic parameters include instantaneous available bandwidth, read / write concurrency, and storage medium type. These parameters are collected using standardized detection commands. Abnormal data is removed during the collection process, and the effective values ​​are calculated after noise reduction. The characteristic protocol association library uses a two-dimensional index structure that combines the native protocol identifier with the device's unique identifier to establish a query index for key characteristic parameter fields and sets a data validity period and a periodic scanning and update mechanism.

3. The method for unified access and management of cloud storage resources combining intelligent gateway and protocol conversion according to claim 2, characterized in that, The storage medium type is coded and classified according to preset rules, the native protocol identifier covers mainstream cloud storage protocol types, the real-time characteristic parameters are bound to the native protocol identifier to form a structured data group, and each data group is accompanied by a timestamp; the detection command is sent through the intelligent gateway transmission interface compatible with general network protocols, and the resource identifier, network address and command sending time information are recorded during the sending process. Resources that do not respond within the timeout period will trigger a retry mechanism or be marked as abnormal resources.

4. The method for unified access and management of cloud storage resources combining intelligent gateway and protocol conversion according to claim 1, characterized in that, In step S2, after receiving a service request, identity authentication and format verification are first completed. For legitimate requests, a unique identifier and the network information of the initiating party are extracted. The operation type is classified according to the data processing behavior, and the data volume is divided into multiple levels according to the total data volume. The total volume of multiple data sets is calculated by accumulating the volume of a single data set. The target resource is screened through a three-dimensional matching model of operation type, data volume, and resource characteristics. The final target resource is determined by combining storage medium compatibility, performance compatibility coefficient, transmission time redundancy, and bandwidth redundancy.

5. The unified access and management method for cloud storage resources combining intelligent gateway and protocol conversion according to claim 4, characterized in that, The performance adaptation coefficient calculation adopts a dynamic weight allocation mechanism, and the weight coefficient is adjusted according to the operation type. When the transmission performance is not up to standard, it is recalibrated based on optimization theory. The customized conversion rules include four core modules: protocol mapping relationship, conversion rate threshold, data fragmentation strategy and format adaptation standard. The protocol mapping establishes a field-level correspondence relationship. When a necessary field is missing, it is automatically supplemented or a retry is triggered. The conversion rate threshold is calculated in combination with the target resource bandwidth carrying capacity. The data fragmentation strategy adapts to the data volume and resource concurrent processing capability. The format adaptation standard matches the storage medium type.

6. The method for unified access and management of cloud storage resources combining intelligent gateway and protocol conversion according to claim 1, characterized in that, Before the protocol conversion is executed in step S3, the protocol parsing library is loaded, the rule syntax is checked and the compatibility is verified. The cache priority is calculated according to the completeness, rationality and compliance of the core rule module, and high-priority rules are cached in memory first. During data transmission, multi-dimensional real-time parameters are collected at a fixed frequency, and after noise reduction processing, they are synchronously updated to the feature protocol association library. Based on the updated data, the conversion rate and data format are dynamically adjusted, and the adjusted rules are applied immediately or with a delay according to their effective priority.

7. The method for unified access and management of cloud storage resources combining intelligent gateway and protocol conversion according to claim 1, characterized in that, Step S4 employs a dual-mode approach of timed detection and event monitoring to monitor protocol changes. It calculates the protocol consistency coefficient by matching the core fields, and determines a protocol change if the coefficient is lower than a preset threshold. It collects four types of interactive features of the changed protocol: version information, field adjustments, authentication methods, and data format requirements. It then re-collects real-time characteristic parameters to generate a new structured data set. The rule generation logic triggers the engine to generate customized conversion rules adapted to the new protocol. The update priority is calculated based on the impact of field changes, the importance of authentication adjustments, and transmission stability requirements to determine when the rules take effect.

8. The method for unified access and management of cloud storage resources combining intelligent gateway and protocol conversion according to claim 1, characterized in that, In step S4, the newly generated associated data and rules are synchronously updated to the feature protocol association library and assigned a unique identifier. The validity is verified by the rule execution error rate. If the error rate exceeds the threshold, a rollback mechanism is triggered to restore the old data and rules from the historical version partition of the association library. At the same time, a graded alarm is issued according to the error rate level. After the data transmission is completed, the transmission quality evaluation value is calculated. The evaluation result is stored in association with the transmission statistics and rule version, taking into account the transmission success rate, rate compliance, and rule adjustment frequency index.

Images