A dynamic expansion method, system and medium of S3 standard object storage interface

Abstract

The present invention discloses a dynamic expansion method, system and medium of an S3 standard object storage interface, and belongs to the field of computer storage technology. The method includes: a system initialization stage; an upload request stage, a double verification is performed on the file upload request, and if the verification is passed, the file upload request is sent to the file processing module; a request processing stage, the file processing module verifies the received file upload request again, and executes different processing procedures according to the type of the file upload request; a flow control and recording stage, flow control and flow recording are performed according to the uploaded flow data; an exception and security event processing stage, during the file upload process, if any module has an internal error, an error report is immediately generated, and exception processing is performed. Through triple verification, while ensuring data security, different processing methods are made according to different request types, thereby improving processing efficiency.

Description

Technical Field

[0001] The present invention relates to the field of computer storage technology, and in particular to a dynamic expansion method, system and medium of an S3 standard object storage interface. Background Art

[0002] With the rapid development of information technology, the demand for data storage and management is growing. As a common data storage method, S3 object storage has been widely used in many fields. However, in actual use, the existing S3 object storage has certain limitations. Traditional S3 object storage is relatively simple in function. Although it provides a basic data storage interface, it cannot directly meet some specific needs, such as batch upload of small files, batch download of files, incremental upload of files, segmented download of single files, package download of files, encrypted storage of files, and file preview. In today's big data era, users often need to process a large number of files. The single storage and download method is inefficient and cannot adapt well to complex and changing business scenarios. Summary of the invention

[0003] The purpose of the present invention is to overcome the deficiencies of the prior art and to provide a method, system and medium for dynamically expanding an S3 standard object storage interface.

[0004] The objective of the present invention is achieved through the following technical solutions: The first aspect of the present invention provides: a dynamic expansion method of the S3 standard object storage interface, comprising the following steps:

[0005] During the system initialization phase, the proxy forwarding module is started and the network monitoring parameters are configured; the signature verification module loads the standard signature verification algorithm library for S3 object storage; the object storage module starts the connection initialization process with the underlying storage device; the file processing module reads the predefined configuration file; the hash calculation module initializes the hash algorithm list; and the traffic control module reads the traffic restriction policy configuration file;

[0006] In the upload request phase, the client initiates a file upload request, and the proxy forwarding module and the signature verification module perform double verification. If the verification passes, the file upload request is sent to the file processing module;

[0007] In the request processing stage, the file processing module re-verifies the received file upload request. After the re-verification is passed, the file processing module executes different processing flows according to the type of file upload request;

[0008] In the flow control and recording stage, when the file processing module is uploading files, the uploaded flow data is read, and flow control and flow recording are performed according to the uploaded flow data;

[0009] During the exception and security incident handling phase, if an internal error occurs in any module during the file upload process, an error report will be generated immediately and exception handling will be performed.

[0010] Preferably, the system initialization stage further includes the following steps:

[0011] Start the proxy forwarding module, configure the network monitoring parameters, including the monitored IP address range, port number, and maximum number of allowed connections, and use the network programming library to start the monitoring service on the specified port to prepare to receive requests from the client;

[0012] The signature verification module loads the standard signature verification algorithm library of the S3 object storage, and reads the pre-configured AK and SK information from the secure key storage medium, and encrypts and stores it in the cache area inside the signature verification module;

[0013] The object storage module starts the connection initialization process with the underlying storage device and checks the status of the underlying storage device, including storage capacity, read and write speed, and availability indicators. If a problem is found in the underlying storage device, the module performs corresponding repairs or alarm operations.

[0014] The file processing module reads the predefined configuration file, which contains the parameter configuration information for batch download, incremental upload, and segment download of files;

[0015] The hash calculation module initializes the hash algorithm list, selects the default hash algorithm according to the system's security requirements and performance requirements, and performs performance optimization settings for the default hash algorithm, including adjustments to the cache size and number of calculation threads;

[0016] The traffic control module reads the traffic restriction policy configuration file, parses the total traffic limit, the single user traffic limit, and the traffic allocation ratio information of different types of requests, and stores this information in an internal data structure.

[0017] Preferably, the upload request stage further includes the following steps:

[0018] The client initiates a file upload request, packages the file data and the request header information including AK and file meta-information according to the request protocol format of the S3 object storage, and sends it to the listening port of the proxy forwarding module through the network;

[0019] After receiving the file upload request, the proxy forwarding module first parses the URL path of the file upload request to determine whether it contains a specific extended capability identification string; if the URL path of the extended capability is not matched, the signature verification step is skipped and the request data is directly forwarded to the object storage module through the pre-established high-speed data channel; if the URL path of the extended capability is matched, the proxy forwarding module forwards the file upload request completely to the signature verification module;

[0020] After the signature verification module receives the file upload request with the extended capability identifier from the proxy forwarding module, it extracts the AK and SK information from the request header information, and performs signature calculation and verification according to the S3 object storage standard signature verification algorithm based on the AK and SK information and the request timestamp, request method, and request resource path in the file upload request; if the signature verification fails, an error information report is generated, which includes the reason for the verification failure, and the error information report is returned to the proxy forwarding module; at the same time, the signature verification module records the security event, including the request source IP address, request time, and request content summary information in the security log file; if the signature verification is successful, the signature verification module returns a response message containing a verification success identifier and a security token to the proxy forwarding module;

[0021] The proxy forwarding module processes the feedback results of the signature verification module. If a response of signature verification failure is received, the proxy forwarding module returns a response message of upload failure to the client, informing the client of the reason for the upload failure; if a response of signature verification success is received, the file upload request carrying the security token of successful signature verification is forwarded to the file processing module.

[0022] Preferably, the request processing stage further includes the following steps:

[0023] After the file processing module receives the file upload request from the proxy forwarding module and the signature verification is successful, it first verifies the validity of the security token carried in the file upload request; if the security token is invalid, the file upload request is rejected and an error message is returned to the proxy forwarding module; if the security token is valid, the processing continues;

[0024] For incremental file upload requests, the file processing module sends a query request to the object storage module to obtain metadata information of the stored target file, including file hash value, file size, and last modification time; the file processing module uses a file difference comparison algorithm to compare the uploaded file with the stored file, divides the uploaded file into blocks of fixed or variable size, calculates the hash value of each block, and compares it with the hash value of the block of the stored file to determine the incremental data block that needs to be uploaded; the file processing module uploads the incremental data block to the object storage through the write interface provided by the object storage module, and during the upload process, calculates the uploaded data volume and upload speed in real time; when the incremental upload is completed, the file processing module updates the metadata information of the file in the object storage, including file size, modification time, and hash value;

[0025] For non-file incremental upload requests, the file processing module directly stores the file data into the object storage through the write interface of the object storage module, and generates and stores the file metadata information at the same time;

[0026] After the file upload of the file incremental upload request or the non-file incremental upload request is completed, the file processing module calls the hash calculation module to calculate the hash value of the file; the hash calculation module uses the selected hash algorithm to perform hash calculation on the file according to the request of the file processing module, and adopts multi-threading or hardware acceleration strategy during the calculation process; the hash calculation module returns the calculated file hash value to the file processing module; the file processing module stores the file hash value together with the file metadata information in the object storage module.

[0027] Preferably, the flow control recording stage further includes the following steps:

[0028] During the file upload process, the file processing module sends traffic data to the traffic control module at a set time interval, including the amount of uploaded data and the current upload rate; after receiving the traffic data from the file processing module, the traffic control module accumulates it into the corresponding traffic statistics counter, and the traffic statistics counter counts the traffic data of different types of file upload requests respectively;

[0029] The flow control module determines whether the upload flow exceeds the limit according to the pre-configured flow restriction strategy and the current flow data; if it does not exceed the limit, no flow control operation is performed, and the file processing module is allowed to upload normally; if it exceeds the limit, the flow control module calculates different flow control parameters according to the flow value after exceeding the limit; the flow control module sends the flow control parameters to the file processing module, and the file processing module adjusts the file upload operation according to the received flow control parameters;

[0030] The traffic control module regularly generates traffic statistics reports, which include the total upload traffic, traffic peak, average traffic, and traffic distribution of different types of upload requests in different time periods. The traffic statistics reports are stored in the database or output as visualization files for administrators to conduct system performance analysis and resource management decisions.

[0031] Preferably, the abnormality and security incident handling stage further includes the following steps:

[0032] During the file upload process, if an internal error occurs in any module, an error report will be generated immediately, including the error type, the location where the error occurred, and the operation context information when the error occurred; the module where the error occurred will send the error report to the system's error processing center, which will classify and process the error according to its severity;

[0033] For minor errors, the error handling center automatically performs repair operations and records error information in the system log file;

[0034] For serious errors, the error handling center will take emergency measures while recording the error information;

[0035] When the signature verification module detects multiple consecutive signature verification failures, it triggers the security protection mechanism while recording the security event and sends a security alert notification to the administrator to inform him of the malicious attack attempt.

[0036] Preferably, the flow control parameters include a ratio for reducing the upload speed and a time interval for suspending upload; when the flow control parameter is a ratio for reducing the upload speed, the file processing module reduces the amount of data stored in the object each time it is written or extends the time interval for writing data; when the flow control parameter is a time interval for suspending upload, the file processing module suspends data transmission and waits for a resume instruction from the flow control module.

[0037] Preferably, the internal errors include memory overflow and disk read and write errors; the minor errors include recoverable memory allocation failures, and the automatic repair operations include reallocating memory and retrying operations; the serious errors include unrecoverable disk failures, and the emergency measures include notifying the administrator, attempting to back up the data being processed to other storage devices, and suspending related service modules; the security protection mechanism includes temporarily blocking the access rights of the IP address, and the blocking time is set according to the security policy.

[0038] The second aspect of the present invention provides: a dynamic expansion system of an S3 standard object storage interface, which is used to implement any of the above-mentioned dynamic expansion methods of the S3 standard object storage interface, comprising:

[0039] The system initialization module is used to start the proxy forwarding module and configure network monitoring parameters; the signature verification module loads the standard signature verification algorithm library of S3 object storage; the object storage module starts the connection initialization process with the underlying storage device; the file processing module reads the predefined configuration file; the hash calculation module initializes the hash algorithm list; the flow control module reads the flow limit policy configuration file;

[0040] The upload request module is used to initiate a file upload request using the client, and the proxy forwarding module and the signature verification module perform double verification. If the verification is passed, the file upload request is sent to the file processing module;

[0041] The request processing module is used to re-verify the received file upload request using the file processing module. After the re-verification is passed, the file processing module executes different processing flows according to the type of the file upload request;

[0042] The flow control and recording module is used to read the uploaded flow data during the process of uploading files by the file processing module, and perform flow control and flow recording according to the uploaded flow data;

[0043] The exception and security event handling module is used to immediately generate an error report and perform exception handling if an internal error occurs in any module during the file upload process.

[0044] The third aspect of the present invention provides: a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are loaded and executed by a processor, any of the above-mentioned dynamic expansion methods of the S3 standard object storage interface is implemented.

[0045] The beneficial effects of the present invention are:

[0046] 1) The triple verification of the proxy forwarding module, signature verification module and file processing module not only ensures data security, but also makes different processing methods according to different request types, thereby improving processing efficiency.

[0047] 2) Functional expansion and efficiency improvement: The functions of S3 object storage have been significantly expanded. New functions such as batch file download, incremental file upload, and single file segment download have greatly improved the flexibility and efficiency of data processing, making data updates more efficient and convenient, effectively improving the speed of the overall business process, and meeting the urgent demand for efficiency in modern big data processing.

[0048] 3) Simplify the development process: Simplify the development process from multiple aspects such as modular design, S3 standard signature verification, and unified interface specifications. Modular design allows developers to focus on a single module and reduce development complexity; standard signature verification reduces the workload of security development; unified interface specifications accelerate function integration and development, improve code maintainability and scalability, and facilitate response to changes in business needs. Developers do not need to make large-scale modifications or re-adaptations to the original system architecture, which greatly reduces implementation costs and technical risks.

[0049] 4) Good compatibility and scalability facilitate subsequent development and integration: The compatibility with the existing S3 object storage system and other related technology ecosystems was fully considered during the design. The core module built based on the S3 object storage standard signature verification method can seamlessly connect to various storage infrastructures and client applications that follow the S3 standard, making it easy to integrate this extension module into the existing S3 application environment. At the same time, the modular design concept of the present invention gives the system excellent scalability. The various functional modules interact with each other through clearly defined interfaces. When new functions need to be added or existing functions need to be upgraded, developers can easily independently develop, replace or expand specific modules without affecting the stability of the overall system. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 A flow chart of a dynamic extension method for the S3 standard object storage interface;

[0051] Figure 2 The flowchart of the file request processing for the dynamic extension method of the S3 standard object storage interface. DETAILED DESCRIPTION

[0052] The technical solution of the present invention will be clearly and completely described below in conjunction with the embodiments. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0053] The present invention adopts golang for module implementation, and the module is strictly constructed based on the S3 object storage standard signature verification method, which fully guarantees the security and integrity of data transmission and storage. Based on the existing rich interfaces of object storage, this extension module can innovatively expand the standard object storage to include file batch download, file incremental upload, single file segment download, file preview and other powerful functions. The S3 object storage extension module of the present invention provides users with more convenience and flexibility in data storage and transmission, meets the diversified needs in different scenarios, and can also expand the object storage of various cloud vendors such as privatized object storage, Baidu Cloud, Huawei Cloud, Tencent Cloud, and Tianyi Cloud, and has broad application prospects. By adopting golang for gateway implementation, the S3 object storage standard signature verification method is strictly followed to ensure the security and integrity of data during transmission and storage, and provide reliable protection for users' data assets. At the same time, innovative extensions are carried out on the basis of the existing rich interfaces of object storage, and the file batch download function is added to meet the needs of users in scenarios such as batch small file upload, file incremental upload, and single file segment download, and improve the stability and adaptability of download.

[0054] See also Figure 1-Figure 2 The first aspect of the present invention provides: a method for dynamically extending the S3 standard object storage interface, comprising the following steps:

[0055] During the system initialization phase, the proxy forwarding module is started and the network monitoring parameters are configured; the signature verification module loads the standard signature verification algorithm library for S3 object storage; the object storage module starts the connection initialization process with the underlying storage device; the file processing module reads the predefined configuration file; the hash calculation module initializes the hash algorithm list; and the traffic control module reads the traffic restriction policy configuration file;

[0056] In the upload request phase, the client initiates a file upload request, and the proxy forwarding module and the signature verification module perform double verification. If the verification passes, the file upload request is sent to the file processing module;

[0057] In the request processing stage, the file processing module re-verifies the received file upload request. After the re-verification is passed, the file processing module executes different processing flows according to the type of file upload request;

[0058] In the flow control and recording stage, when the file processing module is uploading files, the uploaded flow data is read, and flow control and flow recording are performed according to the uploaded flow data;

[0059] During the exception and security incident handling phase, if an internal error occurs in any module during the file upload process, an error report will be generated immediately and exception handling will be performed.

[0060] In this embodiment, the proxy forwarding module is used to forward the newly added capabilities of object storage such as batch file download, incremental file upload, and segmented download of a single file according to the url_path to the signature verification module. The signature verification module is used to verify the signature according to the standard signature verification algorithm of the s3 object storage based on the provided ak and sk. The object storage module is used for s3 object storage, providing uploading, downloading and storage of s3 files. The file processing module is used to upload, download, split, add content, etc. to files with extended capabilities. The hash calculation module is used to calculate the hash value of the file. The flow control module is used to control the transmission flow of the file.

[0061] In some embodiments, the system initialization phase further includes the following steps:

[0062] Start the proxy forwarding module, configure the network monitoring parameters, including the monitored IP address range, port number, and maximum number of allowed connections, and use the network programming library to start the monitoring service on the specified port to prepare to receive requests from the client;

[0063] The signature verification module loads the standard signature verification algorithm library of the S3 object storage, and reads the pre-configured AK and SK information from the secure key storage medium, and encrypts and stores it in the cache area inside the signature verification module;

[0064] The object storage module starts the connection initialization process with the underlying storage device and checks the status of the underlying storage device, including storage capacity, read and write speed, and availability indicators. If a problem is found in the underlying storage device, the module performs corresponding repairs or alarm operations.

[0065] The file processing module reads the predefined configuration file, which contains the parameter configuration information for batch download, incremental upload, and segment download of files;

[0066] The hash calculation module initializes the hash algorithm list, selects the default hash algorithm according to the system's security requirements and performance requirements, and performs performance optimization settings for the default hash algorithm, including adjustments to the cache size and number of calculation threads;

[0067] The traffic control module reads the traffic restriction policy configuration file, parses the total traffic limit, the single user traffic limit, and the traffic allocation ratio information of different types of requests, and stores this information in an internal data structure.

[0068] In this embodiment, a high-performance network programming library is used to open a listening service on a specified port to prepare to receive requests from the client. The signature verification module loads the standard signature verification algorithm library of the S3 object storage to ensure that it can accurately perform signature verification operations. At the same time, the pre-configured AK and SK information is read from the securely configured key storage medium, and encrypted and stored in the cache area inside the module for subsequent rapid retrieval. The underlying storage device with problems is repaired or alarmed accordingly to ensure that the storage module can operate normally. The parameter configuration information of the predefined configuration file read by the file processing module includes the maximum number of concurrent batch downloads, the block size threshold for incremental uploads, the default segment length for segmented downloads, etc., and these parameters are loaded into the memory for subsequent use. The hash calculation module initializes the list of supported hash algorithms, determines the default hash algorithm used according to the security requirements and performance requirements of the system, such as MD5, SHA-256, etc., and performs performance optimization settings on the algorithm, including the adjustment of parameters such as cache size and number of computing threads. The traffic control module reads the traffic restriction policy configuration file, parses the total traffic limit, the traffic limit of a single user, the traffic allocation ratio of different types of requests (upload, download, various extended function requests), and other information, and stores this information in the internal data structure for subsequent traffic monitoring and control.

[0069] In some embodiments, the upload request stage further includes the following steps:

[0070] The client initiates a file upload request, packages the file data and the request header information including AK and file meta-information according to the request protocol format of the S3 object storage, and sends it to the listening port of the proxy forwarding module through the network;

[0071] After receiving the file upload request, the proxy forwarding module first parses the URL path of the file upload request to determine whether it contains a specific extended capability identification string; if the URL path of the extended capability is not matched, the signature verification step is skipped and the request data is directly forwarded to the object storage module through the pre-established high-speed data channel; if the URL path of the extended capability is matched, the proxy forwarding module forwards the file upload request completely to the signature verification module;

[0072] After the signature verification module receives the file upload request with the extended capability identifier from the proxy forwarding module, it extracts the AK and SK information from the request header information, and performs signature calculation and verification according to the S3 object storage standard signature verification algorithm based on the AK and SK information and the request timestamp, request method, and request resource path in the file upload request; if the signature verification fails, an error information report is generated, which includes the reason for the verification failure, and the error information report is returned to the proxy forwarding module; at the same time, the signature verification module records the security event, including the request source IP address, request time, and request content summary information in the security log file; if the signature verification is successful, the signature verification module returns a response message containing a verification success identifier and a security token to the proxy forwarding module;

[0073] The proxy forwarding module processes the feedback results of the signature verification module. If a response of signature verification failure is received, the proxy forwarding module returns a response message of upload failure to the client, informing the client of the reason for the upload failure; if a response of signature verification success is received, the file upload request carrying the security token of successful signature verification is forwarded to the file processing module.

[0074] In this embodiment, the file meta information includes the file name, file size, etc. After receiving the request, the proxy forwarding module first parses the URL path of the request to determine whether it contains a specific extended capability identification string, such as " / batch_upload" for batch file upload, " / incremental_upload" for incremental file upload, etc. If the proxy forwarding module determines that the request is a normal file upload request (that is, the URL path of the extended capability is not matched), the request data (including the file stream and request header information) is directly forwarded to the object storage module through the pre-established high-speed data channel, and the signature verification step is skipped. The reasons for verification failure include signature mismatch, invalid AK, expired timestamp, etc. Record this security event in the security log file for subsequent security auditing and tracking.

[0075] In some embodiments, the request processing stage further includes the following steps:

[0076] After the file processing module receives the file upload request from the proxy forwarding module and the signature verification is successful, it first verifies the validity of the security token carried in the file upload request; if the security token is invalid, the file upload request is rejected and an error message is returned to the proxy forwarding module; if the security token is valid, the processing continues;

[0077] For incremental file upload requests, the file processing module sends a query request to the object storage module to obtain metadata information of the stored target file, including file hash value, file size, and last modification time; the file processing module uses a file difference comparison algorithm to compare the uploaded file with the stored file, divides the uploaded file into blocks of fixed or variable size, calculates the hash value of each block, and compares it with the hash value of the block of the stored file to determine the incremental data block that needs to be uploaded; the file processing module uploads the incremental data block to the object storage through the write interface provided by the object storage module, and during the upload process, calculates the uploaded data volume and upload speed in real time; when the incremental upload is completed, the file processing module updates the metadata information of the file in the object storage, including file size, modification time, and hash value;

[0078] For non-file incremental upload requests, the file processing module directly stores the file data into the object storage through the write interface of the object storage module, and generates and stores the file metadata information at the same time;

[0079] After the file upload of the file incremental upload request or the non-file incremental upload request is completed, the file processing module calls the hash calculation module to calculate the hash value of the file; the hash calculation module uses the selected hash algorithm to perform hash calculation on the file according to the request of the file processing module, and adopts multi-threading or hardware acceleration strategy during the calculation process; the hash calculation module returns the calculated file hash value to the file processing module; the file processing module stores the file hash value together with the file metadata information in the object storage module.

[0080] In this embodiment, adopting a multi-threading or hardware acceleration strategy can improve computing efficiency.

[0081] In some embodiments, the flow control recording stage further includes the following steps:

[0082] During the file upload process, the file processing module sends traffic data to the traffic control module at a set time interval, including the amount of uploaded data and the current upload rate; after receiving the traffic data from the file processing module, the traffic control module accumulates it into the corresponding traffic statistics counter, and the traffic statistics counter counts the traffic data of different types of file upload requests respectively;

[0083] The flow control module determines whether the upload flow exceeds the limit according to the pre-configured flow restriction strategy and the current flow data; if it does not exceed the limit, no flow control operation is performed, and the file processing module is allowed to upload normally; if it exceeds the limit, the flow control module calculates different flow control parameters according to the flow value after exceeding the limit; the flow control module sends the flow control parameters to the file processing module, and the file processing module adjusts the file upload operation according to the received flow control parameters;

[0084] The traffic control module regularly generates traffic statistics reports, which include the total upload traffic, traffic peak, average traffic, and traffic distribution of different types of upload requests in different time periods. The traffic statistics reports are stored in the database or output as visualization files for administrators to conduct system performance analysis and resource management decisions.

[0085] In this embodiment, the regularly generated traffic statistics report includes information such as the total upload traffic volume, traffic peak, average traffic volume, and traffic distribution of different types of upload requests in different time periods (such as the past 1 minute, the past 5 minutes, etc.). The report can be stored in a database or output as a visualization file for administrators to perform system performance analysis and resource management decisions.

[0086] In some embodiments, the abnormality and security incident handling stage further includes the following steps:

[0087] During the file upload process, if an internal error occurs in any module, an error report will be generated immediately, including the error type, the location where the error occurred, and the operation context information when the error occurred; the module where the error occurred will send the error report to the system's error processing center, which will classify and process the error according to its severity;

[0088] For minor errors, the error handling center automatically performs repair operations and records error information in the system log file;

[0089] For serious errors, the error handling center will take emergency measures while recording the error information;

[0090] When the signature verification module detects multiple consecutive signature verification failures, it triggers the security protection mechanism while recording the security event and sends a security alert notification to the administrator to inform him of the malicious attack attempt.

[0091] In this embodiment, when the signature verification module detects multiple consecutive signature verification failures (such as 10 consecutive failures from the same IP address), in addition to recording the security event, it also triggers a security protection mechanism, such as temporarily blocking the access rights of the IP address. The blocking time is set according to the security policy (such as 1 hour), and a security alert notification is sent to the administrator to inform him of possible malicious attack attempts.

[0092] In some embodiments, the flow control parameters include a ratio for reducing the upload speed and a time interval for suspending upload; when the flow control parameter is a ratio for reducing the upload speed, the file processing module reduces the amount of data stored in the object each time it is written or extends the time interval for writing data; when the flow control parameter is a time interval for suspending upload, the file processing module suspends data transmission and waits for the recovery instruction from the flow control module.

[0093] In some embodiments, the internal errors include memory overflow and disk read and write errors; the minor errors include recoverable memory allocation failures, and the automatic repair operations include reallocating memory and retrying operations; the serious errors include unrecoverable disk failures, and the emergency measures include notifying the administrator, attempting to back up the data being processed to other storage devices, and suspending related service modules; the security protection mechanism includes temporarily blocking the access rights of the IP address, and the blocking time is set according to the security policy.

[0094] Efficient modular architecture design: The present invention constructs a modular architecture including proxy forwarding, signature verification, object storage, file processing, hash calculation and flow control. Proxy forwarding intelligently distributes requests based on url_path, separating conventional and extended capability request paths. Signature verification is based on the S3 standard to ensure data transmission security. The file processing module accepts successful verification requests, calls other modules according to predefined logic, and handles various file operations. Each module is independently developed, tested and maintained, interacts through standardized interfaces, reduces coupling, improves development efficiency and system scalability, and ensures stable operation and flexible upgrades of the entire S3 object storage extension function.

[0095] Accurate and optimized signature verification mechanism: A special signature verification module is designed to run strictly in accordance with the S3 object storage standard signature verification algorithm. First, AK and SK in the request are accurately extracted, and then multiple parameters such as request timestamp, request resource path, and request method are combined for in-depth verification. The cache technology is used to store common verification data and intermediate results to reduce repeated calculations and significantly improve the verification speed. At the same time, the verification failures are recorded and classified in detail, and security risk information is fed back in a timely manner to effectively prevent illegal access and data tampering, creating a solid security barrier for the S3 object storage system to ensure the integrity and confidentiality of data during transmission and storage.

[0096] Multifunctional file processing and intelligent traffic control: The file processing module has powerful and diverse functional logic. For incremental file uploads, an efficient file difference comparison algorithm is used to accurately locate new or modified content and reduce data transmission. Batch file downloads use multi-threaded or asynchronous task mode to concurrently process multiple file download tasks, improve download efficiency and integrate packaging. Single-file segmented downloads support breakpoint resumption and flexibly respond to network fluctuations. The traffic control module monitors file transfer traffic in real time and intelligently controls it according to preset strategies, such as limiting total traffic, single user traffic, and allocating traffic ratios of different types of requests, to ensure stable system operation, avoid network congestion, and achieve rational utilization and optimal allocation of resources.

[0097] The second aspect of the present invention provides: a dynamic expansion system of an S3 standard object storage interface, which is used to implement any of the above-mentioned dynamic expansion methods of the S3 standard object storage interface, comprising:

[0098] The system initialization module is used to start the proxy forwarding module and configure network monitoring parameters; the signature verification module loads the standard signature verification algorithm library of S3 object storage; the object storage module starts the connection initialization process with the underlying storage device; the file processing module reads the predefined configuration file; the hash calculation module initializes the hash algorithm list; the flow control module reads the flow limit policy configuration file;

[0099] The upload request module is used to initiate a file upload request using the client, and the proxy forwarding module and the signature verification module perform double verification. If the verification is passed, the file upload request is sent to the file processing module;

[0100] The request processing module is used to re-verify the received file upload request using the file processing module. After the re-verification is passed, the file processing module executes different processing flows according to the type of the file upload request;

[0101] The flow control and recording module is used to read the uploaded flow data during the process of uploading files by the file processing module, and perform flow control and flow recording according to the uploaded flow data;

[0102] The exception and security event handling module is used to immediately generate an error report and perform exception handling if an internal error occurs in any module during the file upload process.

[0103] The third aspect of the present invention provides: a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are loaded and executed by a processor, any of the above-mentioned dynamic expansion methods of the S3 standard object storage interface is implemented.

[0104] The above is only a preferred embodiment of the present invention. It should be understood that the present invention is not limited to the form disclosed herein, and should not be regarded as excluding other embodiments, but can be used in various other combinations, modifications and environments, and can be modified within the scope of the concept described herein through the above teachings or the technology or knowledge of the relevant field. The changes and modifications made by those skilled in the art shall not deviate from the spirit and scope of the present invention, and shall be within the scope of protection of the claims attached to the present invention.

Claims

1. A method for dynamically extending the S3 standard object storage interface, characterized in that: The following steps are involved: During the system initialization phase, the proxy forwarding module is started and the network monitoring parameters are configured; The signature verification module loads the standard signature verification algorithm library of the S3 object storage; the object storage module starts the connection initialization process with the underlying storage device; the file processing module reads the predefined configuration file; The hash calculation module initializes the hash algorithm list; the flow control module reads the flow restriction policy configuration file; In the upload request phase, the client initiates a file upload request, and the proxy forwarding module and the signature verification module perform double verification. If the verification passes, the file upload request is sent to the file processing module; In the request processing stage, the file processing module re-verifies the received file upload request. After the re-verification is passed, the file processing module executes different processing flows according to the type of file upload request; In the flow control and recording stage, when the file processing module is uploading files, the uploaded flow data is read, and flow control and flow recording are performed according to the uploaded flow data; During the exception and security incident handling phase, if an internal error occurs in any module during the file upload process, an error report will be generated immediately and exception handling will be performed; The upload request stage further includes the following steps: The client initiates a file upload request, packages the file data and the request header information containing the AK and file meta-information according to the request protocol format of the S3 object storage, and sends it to the listening port of the proxy forwarding module through the network; After receiving the file upload request, the proxy forwarding module first parses the URL path of the file upload request to determine whether it contains a specific extended capability identification string; if the URL path of the extended capability is not matched, the signature verification step is skipped and the request data is directly forwarded to the object storage module through the pre-established high-speed data channel; if the URL path of the extended capability is matched, the proxy forwarding module forwards the file upload request completely to the signature verification module; After receiving the file upload request with the extended capability identifier from the proxy forwarding module, the signature verification module extracts the AK and SK information from the request header information, and performs signature calculation and verification according to the S3 object storage standard signature verification algorithm based on the AK and SK information and the request timestamp, request method and request resource path in the file upload request; if the signature verification fails, an error information report is generated, which includes the reason for the verification failure, and the error information report is returned to the proxy forwarding module; at the same time, the signature verification module records the security event, including the request source IP address, request time and request content summary information in the security log file; if the signature verification is successful, the signature verification module returns a response message containing a verification success identifier and a security token to the proxy forwarding module; The proxy forwarding module processes the feedback from the signature verification module. If a response indicating that the signature verification failed is received, the proxy forwarding module returns a response message indicating that the upload failed to the client, informing the client of the reason for the upload failure. If a response indicating that the signature verification succeeded is received, the file upload request carrying the security token indicating that the signature verification succeeded is forwarded to the file processing module. The request processing stage further includes the following steps: After the file processing module receives the file upload request from the proxy forwarding module and the signature verification is successful, it first verifies the validity of the security token carried in the file upload request; if the security token is invalid, the file upload request is rejected and an error message is returned to the proxy forwarding module; if the security token is valid, the processing continues; For incremental file upload requests, the file processing module sends a query request to the object storage module to obtain metadata information of the stored target file, including the file hash value, file size and last modification time; the file processing module uses a file difference comparison algorithm to compare the uploaded file with the stored file, divides the uploaded file into blocks of fixed or variable size, calculates the hash value of each block, and compares it with the hash value of the block of the stored file to determine the incremental data block that needs to be uploaded; the file processing module uploads the incremental data block to the object storage through the write interface provided by the object storage module, and during the upload process, calculates the uploaded data volume and upload speed in real time; when the incremental upload is completed, the file processing module updates the metadata information of the file in the object storage, including the file size, modification time and hash value; For non-file incremental upload requests, the file processing module directly stores the file data into the object storage through the write interface of the object storage module, and generates and stores the file metadata information at the same time; After the file upload of the file incremental upload request or the non-file incremental upload request is completed, the file processing module calls the hash calculation module to calculate the hash value of the file; the hash calculation module uses the selected hash algorithm to perform hash calculation on the file according to the request of the file processing module, and adopts multi-threading or hardware acceleration strategy during the calculation process; the hash calculation module returns the calculated file hash value to the file processing module; the file processing module stores the file hash value together with the file metadata information in the object storage module.

2. The dynamic expansion method of the S3 standard object storage interface according to claim 1, characterized in that: The system initialization phase further includes the following steps: Start the proxy forwarding module, configure the network monitoring parameters, including the monitored IP address range, port number and the maximum number of allowed connections, and use the network programming library to start the monitoring service on the specified port to prepare to receive requests from the client; The signature verification module loads the standard signature verification algorithm library of the S3 object storage, and reads the pre-configured AK and SK information from the secure key storage medium, and encrypts and stores it in the cache area inside the signature verification module; The object storage module starts the connection initialization process with the underlying storage device and checks the status of the underlying storage device, including storage capacity, read / write speed, and availability indicators. If a problem is found in the underlying storage device, it performs corresponding repairs or alarm operations. The file processing module reads the predefined configuration file, which contains the parameter configuration information for batch download, incremental upload and segment download of files; The hash calculation module initializes the hash algorithm list, selects the default hash algorithm according to the system's security requirements and performance requirements, and performs performance optimization settings for the default hash algorithm, including adjustment of the cache size and number of calculation threads; The traffic control module reads the traffic restriction policy configuration file, parses the total traffic limit, the single user traffic limit, and the traffic allocation ratio information of different types of requests, and stores this information in an internal data structure.

3. The dynamic expansion method of the S3 standard object storage interface according to claim 1, characterized in that: The flow control recording stage further includes the following steps: During the file upload process, the file processing module sends traffic data to the traffic control module at a set time interval, including the amount of uploaded data and the current upload rate; after receiving the traffic data from the file processing module, the traffic control module accumulates it into the corresponding traffic statistics counter, and the traffic statistics counter counts the traffic data of different types of file upload requests respectively; The flow control module determines whether the upload flow exceeds the limit according to the pre-configured flow restriction strategy and the current flow data; if it does not exceed the limit, no flow control operation is performed, and the file processing module is allowed to upload normally; if it exceeds the limit, the flow control module calculates different flow control parameters according to the flow value after exceeding the limit; the flow control module sends the flow control parameters to the file processing module, and the file processing module adjusts the file upload operation according to the received flow control parameters; The traffic control module regularly generates traffic statistics reports, which include the total upload traffic, traffic peak, average traffic and traffic distribution of different types of upload requests in different time periods. The traffic statistics reports are stored in the database or output as visualization files for administrators to conduct system performance analysis and resource management decisions.

4. The dynamic expansion method of the S3 standard object storage interface according to claim 1, characterized in that: The abnormality and security incident handling stage further includes the following steps: During the file upload process, if an internal error occurs in any module, an error report will be generated immediately, including the error type, the location where the error occurred, and the operation context information when the error occurred; the module where the error occurred will send the error report to the system's error processing center, which will classify and process the error according to its severity; For minor errors, the error handling center automatically performs repair operations and records error information in the system log file; For serious errors, the error handling center will take emergency measures while recording the error information; When the signature verification module detects multiple consecutive signature verification failures, it triggers the security protection mechanism while recording the security event and sends a security alert notification to the administrator to inform him of the malicious attack attempt.

5. The dynamic expansion method of the S3 standard object storage interface according to claim 3, characterized in that: The flow control parameters include the proportion of reducing the upload speed and the time interval for suspending uploading; when the flow control parameter is the proportion of reducing the upload speed, the file processing module reduces the amount of data stored in the object each time it is written or extends the time interval for writing data; when the flow control parameter is the time interval for suspending uploading, the file processing module suspends data transmission and waits for the recovery instruction of the flow control module.

6. The dynamic expansion method of the S3 standard object storage interface according to claim 4, characterized in that: The internal errors include memory overflow and disk read and write errors; the minor errors include recoverable memory allocation failures, and the automatic repair operations include reallocating memory and retrying operations; the serious errors include unrecoverable disk failures, and the emergency measures include notifying the administrator, attempting to back up the data being processed to other storage devices, and suspending related service modules; the security protection mechanism includes temporarily blocking the access rights of the IP address, and the blocking time is set according to the security policy.

7. A dynamic expansion system for the S3 standard object storage interface, characterized in that: A method for implementing the dynamic expansion of the S3 standard object storage interface according to any one of claims 1 to 6, comprising: The system initialization module is used to start the proxy forwarding module and configure network monitoring parameters; the signature verification module loads the standard signature verification algorithm library of S3 object storage; the object storage module starts the connection initialization process with the underlying storage device; the file processing module reads the predefined configuration file; the hash calculation module initializes the hash algorithm list; the flow control module reads the flow limit policy configuration file; The upload request module is used to initiate a file upload request using the client, and the proxy forwarding module and the signature verification module perform double verification. If the verification is passed, the file upload request is sent to the file processing module; The request processing module is used to re-verify the received file upload request using the file processing module. After the re-verification is passed, the file processing module executes different processing flows according to the type of the file upload request; The flow control and recording module is used to read the uploaded flow data during the process of uploading files by the file processing module, and perform flow control and flow recording according to the uploaded flow data; The exception and security event handling module is used to immediately generate an error report and perform exception handling if an internal error occurs in any module during the file upload process.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are loaded and executed by the processor, the dynamic expansion method of the S3 standard object storage interface as described in any one of claims 1-6 is implemented.

Images