Cross-protocol data access method, system and device, storage medium and program

Abstract

The invention provides a cross-protocol data access method, system and device, a storage medium and a program. The method comprises the following steps: receiving a user request sent by a user side, wherein the user request comprises a target file path and an access type of a target file; according to the target file path, the current cache state of the target file is obtained in the metadata process module, the current cache state is used for indicating the storage condition of the target file in a file storage system, and the metadata process module comprises the cache state of each file in a plurality of files; and if the current cache state is a non-cache state, routing to an object storage cluster through an access interface module, and executing a data access operation corresponding to the access type on the target file. According to the method, the data access efficiency can be improved.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a cross-protocol data access method, system, device, storage medium, and program. Background Technology

[0002] File storage organizes data in a directory tree structure, which is suitable for structured data and high-performance small file access (such as model parameter files in model training); while object storage stores unstructured data (such as images, videos, and log files) in a flat structure, and has the advantages of high scalability and low cost.

[0003] In existing technologies, cold data in file storage is migrated to object storage to reduce costs, or data from an object storage cluster is imported into file storage via cloud computing. However, during model training, it is often necessary to clone the dataset from the object storage cluster to file storage to meet training performance requirements, and then archive the results back to the object storage cluster after training to reduce costs. This frequent migration of data between the two protocols results in low data access efficiency. Summary of the Invention

[0004] This application provides a cross-protocol data access method, system, device, storage medium, and program to solve the technical problem of low data acquisition efficiency.

[0005] Firstly, this application provides a cross-protocol data access method, including...

[0006] Receive a user request sent by the user client, the user request including the target file path and access type of the target file;

[0007] Based on the target file path, the current cache state of the target file is obtained in the metadata process module. The current cache state is used to indicate the storage status of the target file in the file storage system. The metadata process module includes the cache state of each file in multiple files.

[0008] If the current cache status is uncached, the access interface module routes to the object storage cluster to perform the data access operation corresponding to the access type on the target file.

[0009] In this embodiment, the complexity of user operations can be reduced, manual switching of protocols or reliance on external tools can be avoided, invalid data retrieval can be reduced, and the efficiency of data access can be improved.

[0010] Optionally, the access interface module routes to the object storage cluster, and performs the data access operation corresponding to the access type on the target file, including:

[0011] The access interface module sends a data retrieval request corresponding to the target file to the object storage cluster.

[0012] Based on the data retrieval request, the target file is obtained from the object storage cluster;

[0013] Perform the data access operation corresponding to the access type on the target file.

[0014] In this embodiment, on-demand retrieval of object storage data can be achieved without requiring manual user intervention in cross-storage data migration, thus avoiding invalid data transmission.

[0015] Optionally, the access interface module sends a data retrieval request corresponding to the target file to the object storage cluster, including:

[0016] The target directory information corresponding to the target file path is obtained through the access interface module;

[0017] From the target directory information, obtain the target object path of the target file in the object storage cluster;

[0018] Based on the target object path, a data retrieval request is generated and sent to the object storage cluster.

[0019] In this embodiment, the path conversion and request construction can be automatically completed through the access interface module, eliminating the need for users to manually configure the object storage access path and reducing operational complexity.

[0020] Optionally, the access type includes a request to read, a request to delete, and a request to update; performing the data access operation corresponding to the access type on the target file includes:

[0021] If the access type is the request read type, the target file is sent to the user terminal through the data sending thread, and the target file is written to the target file path corresponding to the file storage system through the data writing thread. The data sending thread and the data writing thread are parallel tasks.

[0022] Update the current cache status of the target file to the cached status.

[0023] In this embodiment, the user terminal can quickly obtain data, while avoiding repeated fetching of object storage data, which significantly improves the overall efficiency of cross-protocol data access.

[0024] Optionally, based on the data retrieval request, obtaining the target file from the object storage cluster includes:

[0025] Based on the data retrieval request, at least one object storage cluster data corresponding to the target file is obtained;

[0026] Based on the chunked transfer protocol, data processing is performed on the data of the at least one object storage cluster to obtain the target file.

[0027] In this embodiment of the application, the chunked transfer protocol can improve the stability and efficiency of large file retrieval, and enhance the adaptability and reliability of cross-storage data retrieval.

[0028] Optionally, the method further includes:

[0029] Receive incremental events from the object storage cluster;

[0030] The incremental file path corresponding to the incremental event is determined through the metadata process module.

[0031] Based on the incremental file path, a synchronization operation corresponding to the incremental event is triggered in the file storage system.

[0032] In this embodiment, data synchronization between the object storage cluster and the file storage system can be achieved, thereby improving the reliability of data access.

[0033] Optionally, based on the incremental file path, triggering a synchronization operation corresponding to the incremental event in the file storage system includes:

[0034] Determine the target increment type corresponding to the incremental event;

[0035] Obtain the event priority rules, and determine the target priority corresponding to the target increment type based on the event priority rules;

[0036] Based on the target priority corresponding to the incremental event, the processing thread is invoked to execute the synchronization operation corresponding to the incremental event.

[0037] In this embodiment of the application, incremental event subscription can ensure data visibility between file storage and object storage. Figure 1 This is to avoid business interruptions caused by data version mismatch.

[0038] Optionally, based on the target priority corresponding to the incremental event, a processing thread is invoked to execute the synchronization operation corresponding to the incremental event, including:

[0039] Based on the target priority corresponding to the incremental event, the incremental event is updated to the incremental processing queue;

[0040] The processing thread executes the events in the incremental processing queue sequentially until the synchronization operation corresponding to the incremental event is executed.

[0041] In this embodiment of the application, resource allocation is optimized by event priority scheduling, which can ensure the real-time performance of data synchronization, while low-priority events share thread resources, reducing system overhead.

[0042] Optionally, the method further includes:

[0043] If the current cache state is cached, then the access interface module routes to the file storage system to perform the access operation of the target file.

[0044] In this embodiment, fast access to cached files can be achieved, avoiding network latency and protocol adaptation overhead when fetching across storage.

[0045] Secondly, embodiments of this application provide a cross-protocol data access system. The system includes an access interface module, a metadata process module, a task process module, and an access interface module. The access interface module is connected to both the metadata process module and the task process module. The task process module is connected to both the metadata process module and the access interface module.

[0046] The access interface module is used to receive user requests sent by the user terminal, the user requests including the target file path and access type of the target file;

[0047] The metadata process module is used to store the cache status of each file in multiple files, and to obtain the current cache status of the target file from the cache status of each file according to the target file path;

[0048] The task process module is used to, if the current cache state is uncached, route to the object storage cluster through the access interface module to perform data access operations on the target file.

[0049] Thirdly, embodiments of this application provide an electronic device, including: a processor, and a memory communicatively connected to the processor;

[0050] The memory stores computer-executed instructions;

[0051] The processor executes computer execution instructions stored in the memory to implement the method as described in any of the first aspects.

[0052] Fourthly, embodiments of this application provide a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the method described in the first aspect.

[0053] Fifthly, this application provides a computer program product, including a computer program that, when executed by a processor, implements the method described in any of the first aspects.

[0054] The cross-protocol data access method, system, device, storage medium, and program provided in this application can determine the current cache status of a target file through its path. If the current cache status is uncached, the data access operation is directly routed to the object storage cluster via the access interface module. This reduces user complexity, avoids manual protocol switching or reliance on external tools, minimizes invalid data fetching, and improves data access efficiency. Attached Figure Description

[0055] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0056] Figure 1 A schematic diagram illustrating the application scenarios provided in the embodiments of this application;

[0057] Figure 2 A flowchart illustrating a cross-protocol data access method provided in an embodiment of this application;

[0058] Figure 3 A flowchart illustrating another cross-protocol data access method provided in an embodiment of this application;

[0059] Figure 4 A flowchart illustrating another cross-protocol data access method provided in an embodiment of this application;

[0060] Figure 5 This application provides a schematic diagram of an architecture for incremental event subscription processing.

[0061] Figure 6 A schematic diagram of the structure of a cross-protocol data access system provided in this application embodiment;

[0062] Figure 7 A timing diagram illustrating data flow provided in an embodiment of this application;

[0063] Figure 8 A timing diagram illustrating a data fetching process provided in this embodiment of the application;

[0064] Figure 9 A flowchart illustrating a read event provided in an embodiment of this application;

[0065] Figure 10 A flowchart illustrating the event assignment provided in this application embodiment;

[0066] Figure 11 A timing diagram of SEM event processing provided in an embodiment of this application;

[0067] Figure 12 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application.

[0068] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0069] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0070] Figure 1 This is a schematic diagram illustrating an application scenario provided in an embodiment of this application. Please refer to [link / reference]. Figure 1 This application scenario includes client 101 and server 102. A file storage system can be deployed on server 102 to store frequently accessed data. Server 102 is also configured with an object storage cluster to migrate cold data from the file storage system to the object storage cluster.

[0071] The file storage system organizes data in a directory tree structure, which is suitable for structured data and high-performance small file access. When users train models through client 101, the model parameter files in the model training need to be obtained from the file storage system.

[0072] The object storage cluster stores unstructured data in a flat structure. When users train models through client 101, the images, videos, log files, etc. used for model training do not need to be manually retrieved from the object storage cluster. Instead, they initiate a request through the access interface module of the file storage system, and the system automatically determines whether to pull the data from the object storage cluster and cache it in the file storage system before providing access.

[0073] In related technologies, cold data in file storage is migrated to object storage to reduce costs, or data from an object storage cluster is imported into file storage via cloud computing. However, during model training, it is often necessary to clone the dataset from the object storage cluster to file storage to meet training performance requirements, and then archive the results back to the object storage cluster after training to reduce costs. This frequent migration of data between the two protocols results in low data access efficiency.

[0074] The cross-protocol data access method provided in this application receives a user request sent by a user client. The user request includes the target file path and access type of the target file. Based on the target file path, the current cache status of the target file is obtained in the metadata process module. The cache status is used to indicate the storage status of the target file in the file storage system. The metadata process module includes the cache status of each file in multiple files. If the current cache status is uncached, the method routes to the object storage cluster through the access interface module and performs the data access operation corresponding to the access type on the target file.

[0075] During the above execution process, the current cache status of the target file can be determined through the target file path. If the current cache status is uncached, the data access operation is directly routed to the object storage cluster through the access interface module. This reduces user complexity, avoids manually switching protocols or relying on external tools, reduces invalid data fetching, and improves data access efficiency.

[0076] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.

[0077] Figure 2 This is a flowchart illustrating a cross-protocol data access method provided in an embodiment of this application. Please refer to... Figure 2 The method may include:

[0078] S201, Receive user requests sent by the client.

[0079] The execution entity of this application embodiment can be a server or a cross-protocol data access system set on the server. The cross-protocol data access system can be implemented by software or by a combination of software and hardware.

[0080] The system can receive user requests sent by the client through a unified access interface module. User requests can include the target file path and access type of the target file.

[0081] The target file path is the logical access path of the target file in the file storage system. This path has been pre-bound to the storage location of the object storage cluster and is used to locate the cache location of the target file in the file storage system or the actual storage location of the object storage cluster.

[0082] Access types can include read request, delete request, and update request.

[0083] The request read type is used to request to read the target file. It prioritizes reading from the file storage system cache, and if it is not cached, it is pulled from the object storage cluster.

[0084] The request deletion type is used to request the deletion of the target file and its corresponding cached state in the metadata process module of the file storage system. If the original data of the file exists in the object storage cluster, the original data of the target file can be deleted synchronously.

[0085] The request update type is to update the target file. Prioritize updating the cached data in the file storage system, synchronously update the cache status of the metadata process module, and synchronize the updated data to the object storage cluster to overwrite the original data.

[0086] A unified access interface module can be a Portable Operating System Interface for Computing Systems (POSIX). This module can shield the protocol differences between file storage systems and object storage clusters, providing users with a unified file access paradigm without requiring them to manually adapt to the access protocols of different storage systems.

[0087] S202. Based on the target file path, obtain the current cache status of the target file in the metadata process module.

[0088] The current cache status is used to indicate the storage status of the target file in the file storage system.

[0089] The current cache status can include a cached state and an uncached state. A cached state indicates that the complete data of the target file already exists in the file storage system. An uncached state indicates that the target file is not stored in the file storage system, and only the original data exists in the object storage cluster.

[0090] The metadata process module can store the metadata and cache status of each file. The metadata includes at least the target file path, the object ID of the corresponding object storage cluster, the name of the bound object storage bucket, and the cache status update time, which is used to support cache status queries and subsequent cross-storage routing.

[0091] For example, assuming the target file path requested by the user is " / data / train / image_001.jpg", after the access interface module parses the target file path, it sends a query request to the metadata process module. The metadata process module matches the corresponding metadata based on the target file path. The metadata is as follows: "Target file path: / data / train / image_001.jpg; Object bucket name: train-dataset-bucket; Corresponding object ID: image_001.jpg_20251128; Cache status: Uncached; Cache status update time: 2025-11-28 09:00:00; ...". The metadata process module then feeds back the uncached status from this record to the access interface module.

[0092] S203. If the current cache status is uncached, the access interface module will route to the object storage cluster and perform the data access operation corresponding to the access type on the target file.

[0093] The access interface module is an external communication module (EXTC), which can receive routing instructions from the access interface module and establish communication with the object storage cluster through the object storage adaptation layer (e.g., S3 API). It shields the differences between file storage protocols and object storage protocols, enabling cross-storage data interaction.

[0094] In some embodiments, the data retrieval request corresponding to the target file can be sent to the object storage cluster through the access interface module; based on the data retrieval request, the target file can be obtained from the object storage cluster; and data access operations corresponding to the access type can be performed on the target file.

[0095] This application enables on-demand retrieval of object storage data without requiring manual user intervention in cross-storage data migration, thus avoiding invalid data transfer.

[0096] The cross-protocol data access method provided in this application can determine the current cache status of a target file through the target file path. If the current cache status is uncached, the data access operation is directly routed to the object storage cluster through the access interface module. This reduces user operation complexity, avoids manual protocol switching or reliance on external tools, reduces invalid data fetching, and improves data access efficiency.

[0097] Figure 3 This is a flowchart illustrating another cross-protocol data access method provided in an embodiment of this application. Please refer to... Figure 3 The method may include:

[0098] S301, Receive user requests sent by the user terminal.

[0099] S302. Based on the target file path, obtain the current cache status of the target file in the metadata process module.

[0100] The execution process of S301-S302 can be found in the execution process of S201-S202, and will not be repeated here.

[0101] S303. If the current cache status is uncached, send a data retrieval request for the target file to the object storage cluster through the access interface module.

[0102] In some embodiments, the target directory information corresponding to the target file path can be obtained by accessing the interface module; the target object path of the target file in the object storage cluster can be obtained from the target directory information; a data retrieval request can be generated based on the target object path and sent to the object storage cluster.

[0103] The target directory information is the configuration information of the bound directory corresponding to the target file path. The target directory information includes at least the name of the object storage bucket associated with the bound directory, directory mapping rules (such as the binding relationship between the file storage directory and the object storage prefix), data format conventions, etc., which are used to associate the target file with the storage location of the object storage cluster.

[0104] The target object path is the unique identifier path of the target file in the object storage cluster. It is usually composed of "object storage bucket name + object storage prefix + target file name" and is used to accurately locate the target file data in the object storage cluster.

[0105] A data retrieval request is a standardized request initiated by the access interface module to the object storage cluster to request target file data. Its core fields must conform to the interface specifications of the object storage cluster (such as the S3 API specification).

[0106] The access interface module can receive requests forwarded by the access interface module, extract the target file path, and then extract the corresponding parent directory. For example, the target file path is " / data / train / image_002.png", and the parent directory is " / data / train / ".

[0107] The access interface module can match the pre-configured binding directory based on the split parent directory and obtain the corresponding target directory information. This target directory information includes: the name of the bound object storage bucket (such as train-dataset-2025), directory mapping rules, and data format conventions.

[0108] For example, the directory mapping rule can be that the file storage directory " / data / train / " corresponds to the object storage prefix "train / images / "; the data format convention can be "keep the original file format".

[0109] Based on the object bucket name and directory mapping rules in the target directory information, and combined with the original file name of the target file, the target object path is generated by concatenation.

[0110] For example, the original file name of the target file is "image_002.png", and the path of the target object generated by concatenation is "train-dataset-2025 / train / images / image_002.png".

[0111] The access interface module constructs a data retrieval request according to the protocol specifications supported by the object storage cluster. This request includes at least the target object path, data transmission method (such as streaming), and data verification parameters (such as MD5 checksum requirements).

[0112] The data retrieval request is sent to the object storage cluster via the object storage adapter layer.

[0113] For example, assuming the user requests a target file path of " / data / train / image_002.png": The access interface module parses the parent directory of this path as " / data / train / ", matching the corresponding target directory information: object storage bucket name "train-dataset-2025", directory mapping rule " / data / train / → train / images / ", and data format convention "preserve the original file format"; based on the directory mapping rule, the target file path " / data / train / image_002.png" is converted into the target object path "train-dataset-2025 / train / images / image_002.png"; the access interface module constructs a data retrieval request, with the target object storage bucket: train-dataset-2025; target object key: train / images / image_002.png; transmission protocol: HTTPS; data verification method: MD5; the access interface module sends this data retrieval request to the object storage cluster through the S3API client, and waits for the object storage cluster to return the target file data.

[0114] In this application, path conversion and request construction can be automatically completed through the access interface module, eliminating the need for users to manually configure object storage access paths and reducing operational complexity. In addition, standardized data retrieval requests ensure compatibility with object storage clusters, support the connection to object storage services from different vendors, and improve the flexibility and adaptability of cross-protocol data access.

[0115] S304. Based on the data retrieval request, obtain the target file from the object storage cluster.

[0116] In some embodiments, at least one object storage cluster data corresponding to the target file is obtained based on a data retrieval request; the at least one object storage cluster data is processed based on a chunked transfer protocol to obtain the target file.

[0117] After the access interface module sends a data retrieval request to the object storage cluster, the object storage cluster first verifies the validity of the request. After the verification is successful, the object storage cluster returns response information, which includes the total size of the target file, the number of data blocks, the identifier of each data block (e.g., shard ID), and data verification information (e.g., the MD5 value of each data block and the overall MD5 value of the file).

[0118] The access interface module decides whether to use a chunked transfer protocol based on the total size of the target file in the response information: if the target file is a small file, the complete file data is obtained directly in a single transfer; if the target file is a large file (e.g., 5MB / 10MB, which can be adapted to different object storage specifications), the chunked transfer protocol is enabled, and chunked pull requests are sent to the object storage cluster in batches according to the maximum chunk size supported by the object storage cluster (e.g., 5MB / 10MB, which can be adapted to different object storage specifications), and each data block is obtained in turn.

[0119] Small files are defined as those smaller than the block threshold, while large files are defined as those greater than or equal to the block threshold. The block threshold can be adjusted through system configuration; for example, the block threshold can be 100MB.

[0120] After receiving each data block from the object storage cluster, the access interface module first sorts the data blocks according to the shard ID returned by the object storage cluster, and then performs continuous splicing processing according to the splicing rules of the chunk transmission protocol to avoid data block disorder causing file corruption.

[0121] After the concatenation is completed, the access interface module calculates the overall checksum of the concatenated file (e.g., MD5 value) and compares it with the overall checksum of the target file returned by the object storage cluster. If the checksums match, the target file is confirmed to have been successfully retrieved. If the checksums do not match, a retransmission request is initiated for the data blocks that failed the checksum (instead of re-fetching the entire file) until all data blocks pass the checksum before the concatenation is completed, ensuring the integrity of the target file.

[0122] The access interface module can choose whether to use a chunked transfer protocol to perform data retrieval based on requirements. For example, when downloading large files, using a chunked transfer protocol can avoid timeouts or excessive memory usage for single requests, and can also improve transmission efficiency by pulling multiple data blocks in parallel; when downloading small files, a single transfer can be used to reduce the additional overhead of chunking and concatenation, thus balancing the retrieval performance for different file scenarios.

[0123] Chunk transfer protocol refers to a protocol that splits a large file into multiple independent data blocks of fixed size, retrieves each data block separately through multiple requests, and then reassembles them in the original order to restore the complete file after the transfer is completed. This can reduce the risk of timeout and resource consumption during the transfer of large files.

[0124] In at least one object storage cluster data is transmitted in a chunked manner. The target file corresponding to the data block is obtained in batches from the object storage cluster. Each data block contains a chunk ID, data content and its own verification information to ensure that the order is correct and the data is not damaged during splicing.

[0125] For example, suppose the user requests a large file named " / data / train / video_001.mp4" with a total size of 500MB:

[0126] After the access interface module sends a data retrieval request, the object storage cluster returns the following response: total file size 500MB, shard size 10MB, 50 data blocks in total (shard ID from 001 to 050), MD5 value of each shard and MD5 value of the entire file (e.g., abc123def456...).

[0127] The access interface module enables the chunked transfer protocol, which pulls data blocks from the object storage cluster in batches according to the chunk ID order, and supports parallel pulling of 3-5 data blocks (the number of parallel blocks can be configured) to improve the transmission speed.

[0128] For each data block received, the MD5 value of the data block is first checked to see if it matches the corresponding fragment check value returned by the object storage cluster. If the check passes, the data block is temporarily stored; otherwise, the fragment is immediately retransmitted.

[0129] After all 50 data blocks have been retrieved and verified, the data blocks are assembled in the order of fragment ID001 to ID050 to form a complete binary data stream;

[0130] Calculate the MD5 value of the concatenated complete file and compare it with the overall MD5 value abc123def456... returned by the object storage cluster. If they match, the target file video_001.mp4 has been successfully obtained.

[0131] In this application, the dynamic decision-making mechanism and data block verification and retransmission mechanism of the block transfer protocol can improve the stability and efficiency of large file retrieval; it can also adapt to the transmission scenarios of files of different sizes, and improve the adaptability and reliability of cross-storage data retrieval.

[0132] S305. Perform the data access operation corresponding to the access type on the target file.

[0133] In some embodiments, if the access type is a request to read, the target file is sent to the user terminal through a data sending thread, and the target file is written to the target file path corresponding to the file storage system through a data writing thread. The data sending thread and the data writing thread are parallel tasks; the current cache state corresponding to the target file is updated to the cached state.

[0134] In this application, by executing response reading and cache writing in parallel, the client can quickly obtain data, which reduces the latency of the first access and avoids repeatedly fetching object storage data, thus significantly improving the overall efficiency of cross-protocol data access.

[0135] In some embodiments, if the access type is a request to delete, an object deletion request is sent to the object storage cluster to delete the original data corresponding to the target object path; and a status update request is sent to the metadata process module to reset the cached status of the target file to an uncached state, and record the deletion time and deletion reason.

[0136] In some embodiments, if the access type is a request update type, an update operation is performed on the target file based on the update data in the user request, and a data change log is recorded during the update process; a request is sent to the metadata process module to synchronously update metadata such as cache status, file modification time, updated file size, and data verification value to ensure the consistency between metadata and cached files.

[0137] In this application, differentiated processing based on access type improves the flexibility and reliability of cross-protocol data access.

[0138] S306. If the current cache state is cached, then the access interface module will route to the file storage system to perform the access operation of the target file.

[0139] The complete data of the target file is already stored in the file storage system, and the cache status and file information (size, checksum, modification time) in the metadata are consistent with the actual data in the file storage. There is no need to pull data from the object storage cluster; access requests can be responded to directly through the file storage system.

[0140] This application enables fast access to cached files, avoiding network latency and protocol adaptation overhead when fetching files across storage.

[0141] The cross-protocol data access method provided in this application can determine the current cache status of the target file through the target file path. If the current cache status is uncached, the data access operation can be directly routed to the object storage cluster through the access interface module. For cached states, the data can be directly obtained from the file storage system, which can reduce the complexity of user operations, avoid manually switching protocols or relying on external tools, reduce invalid data retrieval, and improve the efficiency of data access.

[0142] It is worth noting that when incremental events of data change occur in the object storage cluster (such as object addition, object content update, object deletion, and object attribute modification), the data changes in the object storage cluster can be synchronized to the metadata process module. The object storage cluster and the file storage system synchronize data in real time, so that users can obtain the latest data when querying, avoiding access errors caused by inconsistencies between the two ends.

[0143] Below, in conjunction with Figure 4 This section provides further explanation of the specific handling process for incremental events in the cross-protocol access method.

[0144] Figure 4 This is a flowchart illustrating another cross-protocol data access method provided in an embodiment of this application. Please refer to... Figure 4 The method may include:

[0145] S401, Receive incremental events from the object storage cluster.

[0146] The access interface module can receive incremental events triggered by the object storage cluster in real time through the incremental event notification mechanism provided by the object storage cluster (such as S3's SQS message queue, object storage's WebHook callback, incremental log subscription, etc.).

[0147] An incremental event must include at least the event type identifier, the target object path, the event trigger time, a data change summary (such as the updated file size, checksum, and the unique identifier of the object to be deleted), and the event source (such as manual user operation or automatic system synchronization).

[0148] For example, when a new file "train-dataset-2025 / train / images / image_003.png" is added to the object storage cluster, an incremental event of type "object addition" is triggered. The incremental event includes the following characteristics: event type is "object addition", target object path is "train-dataset-2025 / train / images / image_003.png", trigger time is "2025-11-29 14:30:00", and data digest is "file size = 2.5MB, MD5 = xyz789…".

[0149] S402. Determine the incremental file path corresponding to the incremental event through the metadata process module.

[0150] The incremental file path refers to the logical access path of the target file corresponding to the incremental event in the file storage system, which is consistent with the target file path format requested by the user.

[0151] The access interface module can extract the target object path from incremental events and separate the object bucket name and object key.

[0152] For example, if the target object path is "train-dataset-2025 / train / images / image_003.png", then the object bucket name is "train-dataset-2025" and the object key is "train / images / image_003.png".

[0153] The split object bucket name and object key are substituted into the binding directory configuration stored in the metadata process module to reverse match the corresponding file storage directory;

[0154] For example, if the binding directory is configured as the file storage directory " / data / train / " ↔ the object storage bucket "train-dataset-2025 + prefix train / images / ", then the object key "train / images / image_003.png" can match the file storage directory " / data / train / ".

[0155] Based on the matched file storage directory, the filename (image_003.png) after the prefix in the object key is concatenated to obtain the incremental file path / data / train / image_003.png.

[0156] The metadata process module queries whether a metadata record already exists for the incremental file path. If it does, it is directly associated with it; otherwise, an initial metadata record is automatically created. The cached state is set to uncached by default, and information such as the associated object storage path is included.

[0157] S403. Based on the incremental file path, trigger the synchronization operation corresponding to the incremental event in the file storage system.

[0158] Synchronization ensures that the cached state of the file data and metadata process modules in the file storage system remains consistent with the incremental changes in the object storage cluster, preventing users from accessing expired data.

[0159] In some embodiments, the target increment type corresponding to the increment event is determined; the event priority rule is obtained, and the target priority corresponding to the target increment type is determined according to the event priority rule; and the processing thread is called according to the target priority corresponding to the increment event to execute the synchronization operation corresponding to the increment event.

[0160] Based on the type identifier of the incremental event, the target incremental type is clearly defined. The target incremental type includes object addition, object update, object deletion, and object attribute modification.

[0161] The metadata process module is pre-configured with event priority rules, which can be set based on the scope of event impact, data access frequency, and business importance.

[0162] For example, high priority is for object updates (involving hot data that users are currently accessing) and object additions (files in frequently accessed directories); medium priority is for object attribute modifications (changes to non-core data such as permissions and descriptions); and low priority is for object deletions (files that are not frequently accessed).

[0163] Based on the target increment type, match the event priority rules to obtain the corresponding target priority.

[0164] Based on the target priority, incremental events are added to the corresponding incremental processing queue, which includes a high-priority queue, a medium-priority queue, and a low-priority queue.

[0165] In this application, incremental event subscription ensures data visibility between file storage and object storage. Figure 1 This is to avoid business interruptions caused by data version mismatch.

[0166] Specifically, based on the target priority corresponding to the incremental event, the incremental event is updated to the incremental processing queue; the processing thread executes the events in the incremental processing queue in sequence until the synchronization operation corresponding to the incremental event is executed.

[0167] Multiple processing threads can be started (the number of threads can be dynamically adjusted, with more threads allocated to high-priority queues). The incremental events are retrieved from the queues in sequence according to the principle of prioritizing high-priority queues and first-in-first-out within the same queue, and the corresponding synchronization operations are executed.

[0168] If the incremental event is an object addition, the access interface module will retrieve the new object data from the object storage cluster according to the incremental file path, write it to the corresponding path in the file storage system, and update the cache status of the metadata process module to the cached state, and synchronize information such as file size and checksum.

[0169] If the incremental event is an object update type, the access interface module will retrieve the updated complete object data, overwrite the cached file at the corresponding path in the file storage system (if the cache does not exist, it will be written directly), and update fields such as file modification time, checksum, and data digest in the metadata to ensure that the cached data is the latest version.

[0170] If the incremental event is an object deletion type, the access interface module sends a deletion request to the file storage system to delete the corresponding cached file (if it exists), resets the cache status of the metadata process module to the uncached state, and records the trigger time and reason of the deletion event.

[0171] If the incremental event is an object property modification, there is no need to operate on the cached data of the file storage system. Instead, the metadata process module updates the attribute fields (such as access permissions and file description) corresponding to the incremental file path to ensure that users obtain the latest attribute information when they access the file.

[0172] In this application, resource allocation is optimized by scheduling events based on priority, which can ensure the real-time nature of data synchronization, while low-priority events (such as log file deletion) share thread resources, reducing system overhead.

[0173] The cross-protocol data access method provided in this application can receive incremental events from an object storage cluster, determine the incremental file path corresponding to the incremental event, and trigger a synchronization operation corresponding to the incremental event in the file storage system based on the incremental file path. This enables data synchronization between the object storage cluster and the file storage system, improving the reliability of data access.

[0174] Figure 5 This is a schematic diagram of an incremental event subscription processing architecture provided for an embodiment of this application. Please refer to [link / reference]. Figure 5 It involves components such as THD_DIAG (diagnostic thread), THD_HDL_n (processing thread), and THD_MGR (management thread), as well as NODE_A (node ​​scheduling unit) and EVENT_A (incremental event core processing node).

[0175] (1) Incremental event triggering and initial flow

[0176] Incremental events are passed from the event notification module of the object storage cluster to the core processing node EVENT_A via “EVENT_C, EVENT_B, EVENT_A”. WAIT_QUE (incremental event waiting queue) is used to temporarily store incremental events to be processed and associate them with EVENT_A. THD_MGR (event management thread) triggers the E1 event, which drives EVENT_A to pull incremental events from the waiting queue and start the processing flow.

[0177] (2) Processing thread interaction with event queue

[0178] THD_HDL_n (incremental event processing thread) corresponds to HDL_QUE_n (incremental processing queue) and HDL_HASH_n (incremental event hash table, used for quick event location), and pushes incremental event processing requests to EVENT_A; NODE_QUE (node ​​scheduling queue) sends node resource requests to NODE_A (node ​​routing unit for incremental events), and NODE_A associates with EVENT_A through E6 events to allocate processing node resources for incremental events (adapting to multi-node cluster scenarios).

[0179] (3) Incremental event processing and status recording

[0180] After EVENT_A completes the incremental event processing, it synchronizes the processing status through the E3 event: on the one hand, it feeds back to NODE_A, and finally associates with the SEMn [1] semaphore through the E7 event, which is used for the synchronization control of singleton resources, such as the concurrency limit of global incremental events; where SEM is an abbreviation for Security Event Management.

[0181] On the other hand, the completed incremental events are pushed to FINISHED_QUE (incremental event completion queue) for subsequent auditing or secondary consumption;

[0182] Meanwhile, EVENT_A is associated with BMP_BLK_A (an incremental event bitmap block used to mark the event processing status) through the E4 event, and then the full information of the incremental event is recorded to three types of archive files through the E5 event (to adapt to the traceability requirements of incremental events):

[0183] Event archiving (SE_FILE: stores the complete content of incremental events, such as event type and object path);

[0184] Progress index archiving (SE_IDX_FILE: records the progress of event processing, supports resuming processing from breakpoints);

[0185] Failure event archive (SE_FAILED_FILE: stores incremental events of processing failures for retries or troubleshooting).

[0186] RULE_BMP (Incremental Event Rule Bitmap, used to match event handling strategies) interacts with EVENT_A through E2 events to match corresponding handling rules for different types of incremental events (such as object addition and deletion).

[0187] (4) Monitoring participation of diagnostic threads

[0188] THD_DIAG (diagnostic thread) sends monitoring requests to NODE_A through DIAG_MODE_QUE (diagnostic mode queue) to obtain information such as the status of incremental event processing nodes and queue backlog in real time, thereby realizing process operation monitoring and fault diagnosis.

[0189] The cross-protocol data access method provided in this application can automatically synchronize data consistency between object storage clusters, file storage systems, and metadata process modules without requiring users to manually trigger updates or verifications. Simultaneously, through priority rules and dynamic thread scheduling, it ensures that incremental events related to core business are processed first, reducing synchronization delays for high-priority data. Furthermore, the synchronization operation is decoupled from the user access process, avoiding the synchronization process from consuming user access resources. This ensures data real-time performance without affecting access efficiency, further improving the reliability and user experience of cross-protocol data access.

[0190] Figure 6 This is a schematic diagram of the structure of a cross-protocol data access system provided in an embodiment of this application. Please refer to... Figure 6 The system includes an access interface module, a metadata process module, a task process module, and an access interface module (EXTC). The access interface module is connected to the metadata process module and the task process module, and the task process module is connected to the metadata process module and the access interface module.

[0191] The access interface module is used to receive user requests sent by the client. The user requests include the target file path and access type of the target file. At the same time, the access interface module supports direct access to file storage and provides file data back to the user.

[0192] The metadata process module is used to store the cache status, topology object clusters and binding relationships of each file in multiple files, and to obtain the current cache status of the target file from the cache status of each file according to the target file path.

[0193] The metadata process module can include two sub-units: task distribution and metadata. The task distribution module receives requests from the task process module and forwards them to the metadata unit. The metadata unit stores and queries information such as the cache status and topology binding relationship of files.

[0194] The task process module is used to route data access operations on the target file to the object storage cluster through the access interface module if the current cache status is uncached.

[0195] The task process module can be divided into two sub-units: a strategy sub-module and an access interface module (EXTC). The strategy sub-module is used to decide whether to trigger an interaction task with the object storage cluster based on the cache status fed back by the metadata process module. The access interface module (EXTC) is used to receive the interaction task with S3 issued by the task process module and to perform data interaction with the object storage cluster, such as pulling object data and synchronizing incremental information.

[0196] Please see Figure 6 The system also includes a page management system, a lower-level management system, and an S3 module. The lower-level management system is connected to the metadata process module, and the S3 module is connected to the object storage cluster and the access interface module, respectively.

[0197] The page management system is used to receive user-defined relationships and policies (such as the binding relationship between file directories and object storage buckets) and distribute the policies to the underlying management system. The underlying management system is used to execute the policies distributed by the page management system and synchronize the topology object cluster and binding relationships to the metadata process module.

[0198] Users do not need to directly manipulate the underlying object storage resource configuration. They can complete policy settings such as binding file directories and object storage buckets through the visual interface of the page management system. This transforms the technical storage resource configuration into business directory association operations that are easy for users to understand, greatly reducing the usage threshold across storage scenarios and enhancing the scalability and flexibility of the system.

[0199] The S3 module is used to receive S3 incremental messages from the object storage cluster and write incremental data to file storage, thereby achieving incremental data synchronization between object storage and file storage.

[0200] To facilitate understanding, the following will be combined with... Figure 7 The data flow corresponding to the cross-protocol data access method provided in the embodiments of this application will be described.

[0201] Figure 7 This is a timing diagram illustrating data flow as provided in an embodiment of this application. Please refer to... Figure 7 This includes the user client, access interface module, metadata process module, access interface module (EXTC), object storage, and file storage. T1-T9 represent time nodes, demonstrating the complete interaction process between the file storage system and object storage. The data flow logic at each time node is as follows:

[0202] T1: Initiate a request. The client sends a file access request (including the target file path) to the access interface module.

[0203] T2: The cache status access interface module sends a request to the metadata process module to query the metadata based on the target file path and determine whether the file is already cached in local file storage or needs to be retrieved from object storage.

[0204] T3: Local cache returned directly (branching scenario). If the metadata shows that the file is cached, the access interface module directly reads the data from the file storage and returns it to the user. This sequence shows the uncached state, which is the branching process.

[0205] T4: Trigger the pass-through read process (uncached scenario). T4: The access interface module sends a pass-through read request to the metadata process module; T5: The metadata process module records the data retrieval request.

[0206] Specifically, T4-1: The metadata process module sends a pass-through read request to the access interface module (EXTC); T4-2: The access interface module initiates a data retrieval request to the object storage; T4-3: The object storage returns a successful retrieval response; T4-4: The access interface module returns the retrieved object data to the metadata process module; T4-5: ​​The metadata process module returns the pass-through read data to the access interface module; T4-6: The access interface module returns the pass-through read data to the user terminal (ensuring that the user terminal can quickly obtain data).

[0207] T5: Background asynchronous fetching and caching (parallel process) metadata process module scans the recorded data fetch requests and sends the formal data fetch request to the access interface module (EXTC).

[0208] T6: Accessing Object Storage to Obtain Complete Data. The access interface module calls the object storage interface to obtain the complete data of the target file and its corresponding metadata.

[0209] T7: Write to file storage to complete the cache access interface module. Convert the retrieved object data into a file format and write it to file storage to complete the local cache.

[0210] T8: The metadata status access interface module returns a "fetch successful (cache write successful)" result to the metadata process module, and the metadata process module updates the cache status of the file to "cached".

[0211] T9: Update the client cache record access interface module to update the local metadata cache record, ensuring that subsequent requests can directly hit the local cache.

[0212] Figure 8 This is a timing diagram illustrating a data fetching process as described in an embodiment of this application. Please refer to [link / reference]. Figure 8This timeline illustrates the background data retrieval interaction process between the metadata process module, the access interface module (EXTC), and the object storage. It completes object data retrieval and metadata synchronization based on topology information. The logic for each time node is as follows:

[0213] T0-0: Read and fetch records. The metadata process module reads the recorded data fetch requests from the local machine (usually fetch tasks generated after a user accesses an uncached file).

[0214] T0-1: Select cluster node. The metadata process module selects the corresponding distributed cluster node (for subsequent distributed synchronization of metadata) based on the configuration of the topology object cluster.

[0215] T1: Initiate a fetch request. The metadata process module sends a data fetch request to the access interface module (EXTC), triggering the object data fetching process.

[0216] T1-0: Request and Response. The access interface module receives the pull request and returns a response to confirm that the task has been received.

[0217] T2: Locate the object storage and pull data. The access interface module locates the corresponding object storage resource based on the pre-bound topology object cluster information (the association between file directories and object storage), and calls the object storage's download_file_object interface to pull the target file data.

[0218] T3: Retrieve Response. After the object storage completes the data preparation, it returns a response to the access interface module to confirm that the data can be retrieved normally (or that data transmission has started).

[0219] T4: Synchronize storage layout. The access interface module synchronizes the storage layout information (such as the file path and shard information in file storage) corresponding to the retrieved object data to the metadata process module.

[0220] T5: Update metadata and unlock. The metadata process module is based on a distributed lock mechanism. It updates the metadata information of the file (such as cache status and storage layout), and then unlocks the metadata resource to ensure that other requests can access the updated metadata normally.

[0221] Figure 9 This is a flowchart illustrating a read event provided in an embodiment of this application. Please refer to [link / reference]. Figure 9This read event is an E1 event. The event management thread periodically replenishes the waiting queue to ensure there are enough events to process, while simultaneously synchronizing the data flow progress. The specific process is as follows: The event management thread initiates the check and replenishment process for read events. The event management thread periodically checks the number of events in the waiting queue. If the number of events is less than a preset threshold (i.e., replenishment is needed), the process proceeds to the next step; otherwise, the process ends directly.

[0222] For all data transfer tasks, calculate the number of events to be read in this batch for each data transfer task based on the gap in the waiting queue and the processing capacity of the data transfer (ensuring that the amount of supplemented events matches the subsequent processing pace). Iterate through all data transfer tasks; if all data transfer events have been read, end the event supplementation process; otherwise, continue execution.

[0223] Switch to the next untraversed data stream task and read one batch of events that meet the required number of records from that data stream. The read events are stored in JSON format. This step parses the core content of the events (such as event type, associated data path, etc.) from the JSON data. The parsed events are then assembled into new nodes in the waiting queue and added to the tail of the waiting queue to replenish the events in the waiting queue.

[0224] Append a progress log (recording the scope of events read this time, supplement time, etc.) to the progress index of the corresponding data flow to facilitate subsequent tracking of the event flow status. The event supplementation process for this batch is complete and awaits the next periodic check.

[0225] Figure 10 This is a flowchart illustrating the event assignment provided in an embodiment of this application. Please refer to [link / reference]. Figure 10 This process describes the allocation logic of events from the waiting queue to each processing queue, ensuring the orderliness of event processing and adaptability to distributed scheduling.

[0226] Upon entering from the starting node, the system first checks if there are "insufficient events in the processing queue?". If insufficient (Y), it retrieves a batch of events from the waiting processing queue (ensuring that events of the same object are processed serially to avoid data conflicts caused by concurrent events of the same object). Next, it assigns a sequence number to each event according to the data flow dimension (this sequence number is used to identify the processing order of events within the same data flow, ensuring that events are executed sequentially according to business logic). Subsequently, the events are distributed to the following three queues / data structures: the event processing queue, the node event queue, and the corresponding data flow hash table.

[0227] The event processing queue is used for subsequent processing threads to directly consume events; the node event queue categorizes events according to the processing node dimension, adapting to task scheduling of distributed nodes; the corresponding data flow hash table: quickly associates events with their respective data flows through the hash structure, facilitating subsequent progress tracking and exception retries.

[0228] If there are enough events (N) in the queue, proceed directly to the end node.

[0229] Figure 11 This is a timing diagram illustrating SEM event processing as provided in an embodiment of this application. Please refer to... Figure 11 This demonstrates the event processing interaction flow between SEMm (master SEM) and SEMs (slave SEMs). SEMm merges events that meet the merging conditions (the events must simultaneously meet the conditions of "event status is waiting" and "belong to the same target node", and the number of messages merged in a single batch cannot exceed 20 to avoid excessive requests causing processing timeouts), and assembles request messages based on the merged events;

[0230] SEMm sends a "request processing" command to SEMs, triggering the event execution flow from the SEMs. After receiving the command, SEMs enters the loop logic, traverses the merged events, and calls the EXTC module to execute the corresponding data flow operations (such as object data retrieval, cache writing, etc.). After completing the event traversal and data flow operations, SEMs returns an "ACK" response to SEMm, informing it that the request has been completed. After receiving the ACK response, SEMm processes the execution result of the data flow (such as updating the event status, recording the execution log, triggering exception retries, etc.).

[0231] Figure 12 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Please refer to... Figure 12 The electronic device 1200 may include a processor 1201 and a memory 1202 communicatively connected to the processor 1201. Exemplarily, the processor 1201 and the memory 1202 are interconnected via a bus 1203.

[0232] Memory 1202 stores computer-executed instructions;

[0233] The processor 1201 executes computer execution instructions stored in the memory 1202, causing the processor 1201 to perform the cross-protocol data access method as described in the above method embodiment.

[0234] Accordingly, embodiments of this application provide a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the cross-protocol data access method of the above-described method embodiments.

[0235] Accordingly, embodiments of this application may also provide a computer program product, including a computer program, which, when executed by a processor, can implement the cross-protocol data access method shown in the above method embodiments.

[0236] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0237] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily essential to this application.

[0238] It should be further noted that although the steps in the flowchart are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowchart may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0239] It should be understood that the above-described device embodiments are merely illustrative, and the device of this application can also be implemented in other ways. For example, the division of units / modules in the above embodiments is only a logical functional division, and there may be other division methods in actual implementation. For example, multiple units, modules, or components may be combined, or integrated into another system, or some features may be ignored or not executed.

[0240] Furthermore, unless otherwise specified, the functional units / modules in the various embodiments of this application can be integrated into one unit / module, or each unit / module can exist physically separately, or two or more units / modules can be integrated together. The integrated units / modules described above can be implemented in hardware or as software program modules.

[0241] When integrated units / modules are implemented in hardware, the hardware can be digital circuits, analog circuits, etc. The physical implementation of the hardware structure includes, but is not limited to, transistors, memristors, etc. Unless otherwise specified, the processor can be any suitable hardware processor, such as a CPU, GPU, FPGA, DSP, and ASIC, etc. Unless otherwise specified, the storage unit can be any suitable magnetic or magneto-optical storage medium, such as Resistive Random Access Memory (RRAM), Dynamic Random Access Memory (DRAM), Static Random Access Memory (SRAM), Enhanced Dynamic Random Access Memory (EDRAM), High-Bandwidth Memory (HBM), Hybrid Memory Cube (HMC), etc.

[0242] If the integrated unit / module is implemented as a software program module and sold or used as an independent product, it can be stored in a computer-readable storage device (CMD). Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a memory and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned memory includes various media capable of storing program code, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard drive, magnetic disk, or optical disk.

[0243] In the above embodiments, the descriptions of each embodiment have their own emphasis. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments. The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as these combinations of technical features do not contradict each other, they should be considered within the scope of this specification.

[0244] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this application are indicated by the following claims.

[0245] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A cross-protocol data access method, characterized in that, include: Receive a user request sent by the user client, the user request including the target file path and access type of the target file; Based on the target file path, the current cache state of the target file is obtained in the metadata process module. The current cache state is used to indicate the storage status of the target file in the file storage system. The metadata process module includes the cache state of each file in multiple files. If the current cache status is uncached, the access interface module routes to the object storage cluster to perform the data access operation corresponding to the access type on the target file.

2. The method according to claim 1, characterized in that, The system routes data to the object storage cluster via the access interface module, and performs data access operations corresponding to the access type on the target file, including: The access interface module sends a data retrieval request corresponding to the target file to the object storage cluster. Based on the data retrieval request, the target file is obtained from the object storage cluster; Perform the data access operation corresponding to the access type on the target file.

3. The method according to claim 2, characterized in that, The access interface module sends a data retrieval request corresponding to the target file to the object storage cluster, including: The target directory information corresponding to the target file path is obtained through the access interface module; From the target directory information, obtain the target object path of the target file in the object storage cluster; Based on the target object path, a data retrieval request is generated and sent to the object storage cluster.

4. The method according to claim 2, characterized in that, The access types include read request type, delete request type, and update request type; Performing the data access operation corresponding to the access type on the target file includes: If the access type is the request read type, the target file is sent to the user terminal through the data sending thread, and the target file is written to the target file path corresponding to the file storage system through the data writing thread. The data sending thread and the data writing thread are parallel tasks. Update the current cache status of the target file to the cached status.

5. The method according to claim 2, characterized in that, Based on the data retrieval request, the target file is obtained from the object storage cluster, including: Based on the data retrieval request, at least one object storage cluster data corresponding to the target file is obtained; Based on the chunked transfer protocol, data processing is performed on the data of the at least one object storage cluster to obtain the target file.

6. The method according to claim 1, characterized in that, The method further includes: Receive incremental events from the object storage cluster; The incremental file path corresponding to the incremental event is determined through the metadata process module. Based on the incremental file path, a synchronization operation corresponding to the incremental event is triggered in the file storage system.

7. The method according to claim 6, characterized in that, Based on the incremental file path, trigger the synchronization operation corresponding to the incremental event in the file storage system, including: Determine the target increment type corresponding to the incremental event; Obtain the event priority rules, and determine the target priority corresponding to the target increment type based on the event priority rules; Based on the target priority corresponding to the incremental event, the processing thread is invoked to execute the synchronization operation corresponding to the incremental event.

8. The method according to claim 7, characterized in that, Based on the target priority corresponding to the incremental event, a processing thread is invoked to execute the synchronization operation corresponding to the incremental event, including: Based on the target priority corresponding to the incremental event, the incremental event is updated to the incremental processing queue; The processing thread executes the events in the incremental processing queue sequentially until the synchronization operation corresponding to the incremental event is executed.

9. The method according to claim 1, characterized in that, The method further includes: If the current cache state is cached, then the access interface module routes to the file storage system to perform the access operation of the target file.

10. A cross-protocol data access system, characterized in that, The system includes an access interface module, a metadata process module, a task process module, and an access interface module. The access interface module is connected to both the metadata process module and the task process module. The task process module is connected to both the metadata process module and the access interface module. The access interface module is used to receive user requests sent by the user terminal, the user requests including the target file path and access type of the target file; The metadata process module is used to store the cache status of each file in multiple files, and to obtain the current cache status of the target file from the cache status of each file according to the target file path; The task process module is used to, if the current cache state is uncached, route to the object storage cluster through the access interface module to perform data access operations on the target file.

11. An electronic device, characterized in that, include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory to implement the method as described in any one of claims 1-9.

12. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions that, when executed by a processor, are used to implement the method described in any one of claims 1-9.

13. A computer program product, characterized in that, Includes a computer program that, when executed by a processor, implements the method described in any one of claims 1-9.

Images