Cross-platform file transfer method and apparatus
By working together with the protocol conversion module and the storage intermediary layer, the problem of poor protocol compatibility in cross-platform file transfer is solved, achieving efficient and reliable heterogeneous data interaction, reducing hardware resource consumption, and improving transmission efficiency and consistency.
Patent Information
- Application Number
- CN202611389629.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-09-08
- Publication Date
- 2026-11-17
AI Technical Summary
Existing technologies lack a unified heterogeneous storage intermediary coordination mechanism for cross-platform file transfer, resulting in poor protocol compatibility, high resource overhead, and difficulty in achieving efficient heterogeneous data interaction. In particular, traditional solutions cannot guarantee high availability in scenarios with high requirements for data consistency and security.
By adopting a protocol decoupling and unified abstraction approach, the protocol conversion module parses file operation requests, generates standardized operation instructions, and dynamically determines the target storage path and type by querying the metadata mapping table through the storage intermediary layer, and selects the target storage adapter to achieve cross-platform file transfer.
It achieves protocol compatibility and data consistency for cross-platform file transfer, reduces hardware resource consumption, improves transmission efficiency and availability, and solves the problems of poor compatibility and resource waste caused by the lack of a unified coordination mechanism in existing technologies.
Smart Images

Figure CN123123333A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of cloud computing technology or other related fields, and more specifically, to a cross-platform file transfer method and apparatus. Background Technology
[0002] In existing technologies, as enterprise IT (Information Technology) architectures evolve towards hybrid cloud and heterogeneous storage environments, the demand for file sharing between different operating systems is increasing, and two main solutions are direct protocol conversion or dual protocol stack parallelism.
[0003] While direct protocol conversion can achieve protocol interoperability, it requires the deployment of dedicated intermediate equipment, which not only poses a single point of failure risk but also limits the conversion efficiency to meet the needs of large-scale data transmission due to equipment performance limitations. Dual protocol stack parallel solutions require the simultaneous maintenance of service components in the storage system, resulting in a 30% to 50% increase in hardware resource consumption. Furthermore, each module accesses the underlying storage independently, lacking unified coordination, which can easily lead to data inconsistency or loss.
[0004] Furthermore, existing technologies often lack a unified intermediary layer to shield underlying storage differences when handling cross-platform file transfers. This leads to complex protocol adaptation, high resource consumption, and long response times. Moreover, the lack of an effective metadata caching mechanism results in wasted bandwidth when repeatedly transferring the same file, failing to fully utilize the performance advantages of different storage media. Especially in scenarios with extremely high requirements for data consistency and security, such as finance, traditional solutions struggle to achieve efficient heterogeneous data interaction while ensuring high availability.
[0005] There is currently no effective solution to the above problems. Summary of the Invention
[0006] The main objective of this application is to provide a cross-platform file transfer method and apparatus to at least solve the technical problem of poor protocol compatibility in cross-platform file transfer caused by the lack of a unified heterogeneous storage intermediary coordination mechanism in related technologies.
[0007] To achieve the above objectives, according to one aspect of this application, a cross-platform file transfer method is provided. The method includes: receiving a file operation request from a client with a heterogeneous operating system, wherein the file operation request is generated based on a source file system protocol; parsing the file operation request through a protocol conversion module, extracting file data and metadata from the file operation request, and generating standardized operation instructions based on the file data and metadata; sending the standardized operation instructions to a storage intermediary layer, whereby the storage intermediary layer queries a metadata mapping table to determine the physical storage path and storage type of the target storage system, and selects a target storage adapter based on the storage type; and performing a cross-platform transfer operation on the file data in the standardized operation instructions and the target storage system through the target storage adapter based on the physical storage path, obtaining the execution result and feeding it back to the client.
[0008] Further, the step of generating standardized operation instructions based on the file data and the metadata includes: parsing source file attribute information from the metadata, wherein the source file attribute information includes: permission identifier and creation time information of the source operating system; converting the source file attribute information into target file attribute information, wherein the target file attribute information includes: a target permission model compatible with the target storage system and a unified format timestamp; parsing the target operation type of the file operation request; and encapsulating the file data, the target file attribute information, and the target operation type into the standardized operation instructions.
[0009] Further, the step of querying the metadata mapping table by the storage intermediary layer to determine the physical storage path and storage type of the target storage system, and selecting a target storage adapter based on the storage type, includes: extracting file type identifiers from the metadata; querying the metadata mapping table based on the file type identifiers to obtain at least one candidate storage node corresponding to the file type identifier, thus obtaining a candidate storage node list; calculating the load weight of each candidate storage node, and selecting a target storage node from the candidate storage node list based on the load weights; and loading the target storage adapter corresponding to the target storage node according to the storage type.
[0010] Furthermore, the step of determining the physical storage path and storage type of the target storage system by querying the metadata mapping table by the storage intermediary layer further includes: querying the metadata mapping record in the local cache to obtain a first query result; if the first query result is empty, querying the metadata mapping record in the distributed metadata database to obtain a second query result; writing the metadata mapping record queried in the first query result or the second query result into the local cache and setting the cache expiration time; and determining the physical storage path and the corresponding storage type based on all the queried metadata mapping records.
[0011] Furthermore, the step of performing cross-platform transmission operations on the file data in the standardized operation instruction and the target storage system based on the physical storage path includes: when the target operation indicated by the standardized operation instruction is a write operation, dividing the file data into at least one data fragment; sending the at least one data fragment in parallel to different storage nodes of the target storage system for storage based on the physical storage path; generating a write success confirmation result after receiving storage success signals for all data fragments; and triggering a rollback operation and generating a write failure confirmation result if storage success signals for all data fragments are not received within a preset time window.
[0012] Furthermore, the step of performing cross-platform transmission operations on the file data in the standardized operation instruction and the target storage system based on the physical storage path further includes: when the target operation indicated by the standardized operation instruction is a read operation, determining at least one data fragment location in the target storage system according to the physical storage path; acquiring at least one data fragment in parallel from the at least one data fragment location through a parallel data channel; performing integrity verification on the acquired at least one data fragment, and reassembling the at least one data fragment into complete file data after the integrity verification passes.
[0013] Furthermore, the step of obtaining the execution result and feeding it back to the client includes: encrypting the execution result; and sending the encrypted execution result to the client through a secure transmission channel, wherein the secure transmission channel is established using a transport layer security protocol.
[0014] To achieve the above objectives, according to another aspect of this application, a cross-platform file transfer device is also provided. The device includes: a receiving unit for receiving a file operation request from a client with a heterogeneous operating system, wherein the file operation request is generated based on a source file system protocol; a parsing unit for parsing the file operation request through a protocol conversion module, extracting file data and metadata from the file operation request, and generating standardized operation instructions based on the file data and metadata; a sending unit for sending the standardized operation instructions to a storage intermediary layer, whereby the storage intermediary layer queries a metadata mapping table to determine the physical storage path and storage type of the target storage system, and selects a target storage adapter based on the storage type; and a feedback unit for performing a cross-platform transfer operation on the file data in the standardized operation instructions and the target storage system through the target storage adapter, based on the physical storage path, obtaining the execution result, and feeding it back to the client.
[0015] Further, the parsing unit includes: a first parsing module, used to parse source file attribute information from the metadata, wherein the source file attribute information includes: permission identifier and creation time information of the source operating system; a conversion module, used to convert the source file attribute information into target file attribute information, wherein the target file attribute information includes: a target permission model compatible with the target storage system and a unified format timestamp; a second parsing module, used to parse the target operation type of the file operation request; and an encapsulation module, used to encapsulate the file data, the target file attribute information, and the target operation type into the standardized operation instruction.
[0016] Further, the sending unit includes: an extraction module for extracting file type identifiers from the metadata; a first acquisition module for querying the metadata mapping table based on the file type identifiers to obtain at least one candidate storage node corresponding to the file type identifier, thereby obtaining a candidate storage node list; a calculation module for calculating the load weight of each candidate storage node and selecting a target storage node from the candidate storage node list based on the load weights; and a loading module for loading the target storage adapter corresponding to the target storage node according to the storage type.
[0017] Furthermore, the sending unit further includes: a first query module, used to query metadata mapping records in the local cache to obtain a first query result; a second query module, used to query metadata mapping records in the distributed metadata database to obtain a second query result when the first query result is empty; a writing module, used to write the metadata mapping records queried in the first query result or the second query result into the local cache, and set the cache expiration time; and a first determining module, used to determine the physical storage path and the corresponding storage type based on all the queried metadata mapping records.
[0018] Furthermore, the feedback unit includes: a partitioning module, used to partition the file data into at least one data fragment when the target operation indicated by the standardized operation instruction is a write operation; a storage module, used to send the at least one data fragment in parallel to different storage nodes of the target storage system for storage based on the physical storage path; a generation module, used to generate a write success confirmation result after receiving storage success signals for all data fragments; and a triggering module, used to trigger a rollback operation and generate a write failure confirmation result if storage success signals for all data fragments are not received within a preset time window.
[0019] Furthermore, the feedback unit further includes: a second determining module, configured to determine at least one data fragment location in the target storage system based on the physical storage path when the target operation indicated by the standardized operation instruction is a read operation; a second acquiring module, configured to acquire at least one data fragment in parallel from the at least one data fragment location through a parallel data channel; and a reassembly module, configured to perform integrity verification on the acquired at least one data fragment, and reassemble the at least one data fragment into complete file data after the integrity verification passes.
[0020] Furthermore, the feedback unit also includes: an encryption module for encrypting the execution result; and a sending module for sending the encrypted execution result to the client through a secure transmission channel, wherein the secure transmission channel is established using a transport layer security protocol.
[0021] To achieve the above objectives, according to another aspect of this application, a computer-readable storage medium is also provided, the computer-readable storage medium including a stored computer program, wherein, when the computer program is executed, it controls the device where the computer-readable storage medium is located to perform the cross-platform file transfer method described in any one of the above claims.
[0022] To achieve the above objectives, according to another aspect of this application, an electronic device is also provided, including one or more processors and a memory, the memory being used to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors cause the one or more processors to implement the cross-platform file transfer method described in any of the above claims.
[0023] To achieve the above objectives, according to another aspect of this application, a computer program product is also provided, including computer instructions, wherein when the computer instructions are executed by a processor, they implement the steps of the cross-platform file transfer method described in any of the above claims.
[0024] This invention proposes a cross-platform file transfer method. First, it receives a file operation request from a client with a heterogeneous operating system. This request is generated based on the source file system protocol. Then, a protocol conversion module parses the file operation request, extracting the file data and metadata. Based on this data, standardized operation instructions are generated and sent to a storage intermediary layer. The intermediary layer queries a metadata mapping table to determine the physical storage path and storage type of the target storage system. Based on the storage type, a target storage adapter is selected. Finally, through the target storage adapter, a cross-platform transfer operation is performed on the file data in the standardized operation instructions and the target storage system, based on the physical storage path. The execution result is then obtained and fed back to the client.
[0025] This invention employs a "protocol decoupling and unified abstraction" approach, utilizing a dual processing method involving a "protocol conversion module" and a "storage intermediary layer." First, the protocol conversion module parses file operation requests generated based on the source file system protocol, extracts file data and metadata, and generates standardized operation instructions, thus masking the heterogeneous differences in protocol formats across different operating systems. Subsequently, the storage intermediary layer queries a metadata mapping table based on the metadata to dynamically determine the physical storage path and storage type of the target storage system, and accurately matches the corresponding target storage adapter, achieving flexible mapping between logical instructions and physical storage resources. This process converts cross-platform file transfer requests into a unified internal format and performs intelligent routing, thereby achieving complete decoupling between the client and the underlying heterogeneous storage system. It eliminates compatibility barriers caused by differences in protocol versions, authentication mechanisms, and file attribute definitions, and solves the technical problem of poor protocol compatibility in cross-platform file transfer caused by the lack of a unified heterogeneous storage intermediary coordination mechanism in related technologies. Attached Figure Description
[0026] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments and descriptions of this application are used to explain this application and do not constitute an undue limitation of this application. In the drawings:
[0027] Figure 1 A hardware structure block diagram of a computer terminal (or mobile device) for implementing a cross-platform file transfer method is shown.
[0028] Figure 2 This is a flowchart of an optional cross-platform file transfer method according to an embodiment of the present invention;
[0029] Figure 3 This is a schematic diagram of an optional cross-platform file transfer device according to an embodiment of the present invention;
[0030] Figure 4 This is a structural block diagram of an electronic device that performs a cross-platform file transfer method according to an embodiment of the present invention. Detailed Implementation
[0031] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.
[0032] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.
[0033] To facilitate understanding of the present invention by those skilled in the art, some terms or nouns involved in the various embodiments of the present invention are explained below:
[0034] SMB, Server Message Block, is an application-layer network file-sharing protocol primarily used for sharing files, printers, and serial ports within a local area network (LAN). It allows client applications to access files on a server as if they were local files.
[0035] NFS, Network File System, is a distributed file system protocol that allows client systems to access files and data on remote servers over a network as if they were accessing a local disk. It provides cross-platform file sharing capabilities and enables efficient, low-overhead file data reading, writing, and sharing in clusters, big data storage, and development and testing environments.
[0036] It should be noted that the cross-platform file transfer method and apparatus in this application can be used in the field of cloud computing technology for unified management or data interaction and access of heterogeneous storage resources or distributed storage clusters, and can also be used in any field other than cloud computing technology for unified management or data interaction and access of heterogeneous storage resources or distributed storage clusters. This application does not limit the application field of the cross-platform file transfer method and apparatus.
[0037] It should be noted that all relevant information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, and displayed data) involved in this application are information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, storage, processing, transmission, provision, disclosure, use, and handling of such data comply with the laws, regulations, and standards of the relevant regions, necessary confidentiality measures have been taken, and they do not violate public order and good morals. Corresponding access points are provided for users to choose to authorize or refuse access. For example, this system has interfaces with relevant users or organizations. Before obtaining relevant information, a request to obtain the information needs to be sent to the aforementioned user or organization through the interface, and the relevant information is obtained only after receiving consent from the aforementioned user or organization.
[0038] The information collection (e.g., user voice, video, and text collection) and analysis operations involved in this application have provided users with corresponding operation entry points during execution, allowing users to choose to agree to or reject the automated decision results; if the user chooses to reject, the process will proceed to the expert decision-making process.
[0039] The following embodiments of the present invention can be applied to various systems / applications / devices that require cross-operating system platform file sharing and unified data access from heterogeneous storage systems, enabling a highly compatible and highly available cross-platform file transfer service. The present invention uses a protocol conversion module that works in conjunction with a storage intermediary layer to parse and standardize instructions for source file system protocols (such as SMB / CIFS or NFS). Then, it dynamically routes the data to the target storage adapter via a metadata mapping table to perform underlying storage operations. This better shields the protocol differences and resource heterogeneity of the underlying heterogeneous storage systems, achieving efficient and consistent file transfer between different operating systems, and reducing dependence on dedicated conversion equipment and hardware resource overhead.
[0040] The present invention will now be described in detail with reference to various embodiments.
[0041] Example 1
[0042] According to an embodiment of the present invention, an embodiment of a cross-platform file transfer method is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.
[0043] The cross-platform file transfer method embodiment provided in Embodiment 1 of the present invention can be executed on a mobile terminal, computer terminal or similar computing device. Figure 1 A hardware block diagram of a computer terminal (or mobile device) for implementing a cross-platform file transfer method is shown. Figure 1 As shown, the computer terminal 10 (or mobile device) may include one or more processors 102 (shown as 102a, 102b, ..., 102n in the figure) 102 (processor 102 may include, but is not limited to, a microprocessor MCU or a programmable logic device FPGA, etc.), a memory 104 for storing data, and a transmission device 106 for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of a BUS bus), a network interface, a power supply, and / or a camera. Those skilled in the art will understand that... Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the aforementioned electronic device. For example, computer terminal 10 may also include... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown.
[0044] It should be noted that the aforementioned one or more processors 102 and / or other data processing circuits are generally referred to herein as "data processing circuits". These data processing circuits may be embodied, in whole or in part, in software, hardware, firmware, or any other combination thereof. Furthermore, the data processing circuits may be a single, independent processing module, or may be integrated, in whole or in part, into any other element within the computer terminal 10 (or mobile device). As involved in the embodiments of this application, the data processing circuits serve as a processor control mechanism (e.g., selection of a variable resistor termination path connected to an interface).
[0045] The memory 104 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the cross-platform file transfer method in this embodiment. The processor 102 executes various functional applications and data processing by running the software programs and modules stored in the memory 104, thereby realizing the aforementioned cross-platform file transfer method. The memory 104 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to the computer terminal 10 via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.
[0046] The transmission device 106 is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by the communication provider of the computer terminal 10. In one example, the transmission device 106 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 106 may be a Radio Frequency (RF) module, used for wireless communication with the Internet.
[0047] The display may be, for example, a touchscreen liquid crystal display (LCD) that allows the user to interact with the user interface of the computer terminal 10 (or mobile device).
[0048] Under the above operating environment, the present invention provides, as follows: Figure 2The cross-platform file transfer method shown is implemented by a cross-platform file transfer system. Combining protocol conversion and metadata mapping routing technologies, it is used for file data interaction and transfer between heterogeneous operating systems through a unified intermediary layer. Specifically, it addresses the technical problems of poor protocol compatibility, low data interaction efficiency between heterogeneous storage systems, and data inconsistency caused by the lack of a unified coordination mechanism in existing cross-platform file transfer technologies. This is achieved by standardizing source-end protocol requests and dynamically mapping them to target storage resources. Specifically, it receives file operation requests generated based on the source file system protocol, extracts file data and metadata through a protocol conversion module to generate standardized operation instructions, and uses a storage intermediary layer to query a metadata mapping table to determine the physical storage path and storage type of the target storage system and select the corresponding target storage adapter. Finally, the adapter executes the cross-platform transfer operation based on the physical storage path and returns the result. This achieves the goals of shielding underlying storage differences, enabling efficient and consistent cross-platform file transfer, and decoupling the client from the underlying storage system.
[0049] The embodiments of the present invention will now be described in detail with reference to each specific step.
[0050] Figure 2 This is a flowchart of an optional cross-platform file transfer method according to an embodiment of the present invention, such as... Figure 2 As shown, the method includes the following steps:
[0051] Step S201: Receive a file operation request from a client with a heterogeneous operating system, wherein the file operation request is generated based on the source file system protocol.
[0052] It should be noted that heterogeneous operating systems refer to operating system environments with different kernel architectures, file system specifications, and permission management models. In this embodiment of the invention, heterogeneity is mainly manifested in the differences between operating systems. For example, the first type of operating system may use the NTFS file system and the SMB / CIFS protocol for network file sharing. Its permission model is based on ACL (Access Control List) and SID (Security Identifier), and it has specific requirements for file size and character encoding (such as UTF-16). The second type of operating system may use file systems such as ext4, XFS, or HDFS, and use the NFS protocol or local file system interface. Its permission model is based on the POSIX standard UID / GID (User / Group ID) and rwx permission bits, and the character encoding is mostly UTF-8. Due to the fundamental differences between the two in underlying storage abstraction, metadata definition (such as timestamp precision and special attributes), and network protocol stack, direct cross-platform access will lead to problems such as loss of permissions, garbled characters, or inability to mount.
[0053] A client refers to the terminal device or application process that initiates a file transfer or access request; it is the implementer of the source file system protocol. For the first type of operating system, the first type of client can be a built-in file explorer (accessed via mapped network drives), an application, or an SMB client service running on that operating system. For the second type of operating system, the second type of client can be an application process on the server, a command-line tool (such as cp, mv, scp, etc., which call NFS or local interfaces at the underlying level), or a mount point that mounts the NFS file system. It's important to note that the client does not directly perceive the physical form of the backend storage (whether it's object storage, block storage, or HDFS); it only cares about the interface semantics defined by the source protocol (SMB or NFS). For example, a first-type client thinks it is operating on a "shared folder," while a second-type client thinks it is operating on an "NFS mount point."
[0054] File operation requests refer to specific instruction data packets encapsulated by the client according to the source file system protocol specifications, used for reading, writing, or managing file attributes. The request content includes: 1. Control information: operation type (open, close, read, write, create directory, delete, get attributes, etc.), file handle or path identifier, user authentication token (Token / Session Key). 2. Data payload: for write operations, it includes the binary data stream to be written; for read operations, it may include offset and length. 3. Metadata: filename, extension, permission requests (e.g., read-only, executable), timestamp requests, owner information, file size, etc. It should be noted that the metadata carried in the request often reflects characteristics of the source system. For example, an SMB WRITE request on a first-type operating system may contain specific file locking information, while an NFS WRITE request on a second-type operating system may not contain complex locking mechanisms or use different attribute flags (e.g., setuid / setgid bits).
[0055] The source file system protocol refers to the standard network file sharing protocol followed by the communication between the client and the protocol conversion module, which defines the format, interaction flow and semantics of requests and responses.
[0056] When transferring files between heterogeneous operating systems, significant differences in semantics, data format, and state management between the source file system protocols (SMB and NFS) used by the client make direct interoperability difficult. Existing technologies lack a unified intermediary layer to understand and convert these differences, resulting in poor protocol compatibility, metadata loss (such as incorrect mapping of permissions and timestamps), and low data transmission efficiency. This invention introduces a protocol conversion module to parse and convert file operation requests generated by different source protocols into internally common standardized operation instructions, thereby shielding the underlying heterogeneity and achieving seamless cross-platform file sharing.
[0057] Step S202: The file operation request is parsed by the protocol conversion module, the file data and metadata in the file operation request are extracted, and standardized operation instructions are generated based on the file data and metadata.
[0058] It should be noted that the protocol conversion module is the core processing unit of this invention, located between the client and the storage intermediary layer. It is responsible for semantic mapping and instruction conversion between different network file system protocols. This module is not a simple protocol pass-through, but a middleware component with deep parsing and reconstruction capabilities. This module mainly consists of two sub-parts: 1. Protocol parser: responsible for disassembling data packets of source protocols such as SMB or NFS, stripping the protocol header, trailer, and specific control fields, and extracting pure file operation semantics. 2. Instruction mapping engine: responsible for converting the parsed source semantics into internally common standardized instructions, used to maintain a mapping table between source protocol features and target storage operations.
[0059] In terms of functionality, the protocol conversion module supports dynamic identification and processing of multiple protocol versions, including SMB 1.0 / 2.0 / 3.0 and NFS v3 / v4. It can load corresponding parsing rules based on the protocol version of the client connection and also has bidirectional conversion capabilities, supporting conversion from the first type of operating system (SMB) to the second type of operating system (NFS / local) and vice versa, ensuring consistency in bidirectional transmission. Most importantly, it features decoupling; the protocol conversion module is decoupled from the specific storage backend, focusing only on protocol-level instruction standardization and not directly operating the underlying storage devices. It communicates with the storage intermediary layer via gRPC.
[0060] File operation requests are interactive data packets initiated by the client and conforming to a specific source file system protocol specification. They serve as the trigger signal for the entire transmission process. Request types include, but are not limited to, file read (READ), file write (WRITE), file create (CREATE), directory traversal (LIST), attribute retrieval (GETATTR), permission modification (SETATTR), and file deletion (DELETE). The data structure of a file operation request strictly adheres to the SMB or NFS protocol standard (source protocol). For example, an SMB request contains a specific PDU (Protocol Data Unit) structure, including identifiers such as Session ID, Tree ID, and FileID; an NFS request is based on RPC (Remote Procedure Call) and includes the procedure number, version number, and procedure number. The file operation request must also carry context, including the current session state, user identity context, and the current state information of the file system.
[0061] After parsing the file operation request, the protocol conversion module extracts two core types of information: file data and metadata, which together constitute the complete file transfer content. File data refers to the file's content itself, i.e., the payload composed of byte sequences. For large files, the data needs to be fragmented (default fragment size is 4MB) to facilitate parallel transmission and cache management. Data integrity must be guaranteed during transmission, usually accompanied by a checksum mechanism. Whether it's SMB or NFS, file data is essentially a binary stream, but the transmission encapsulation format differs. The protocol conversion module needs to uniformly extract it into a raw data stream or a standardized data block sequence. Furthermore, metadata refers to a set of information describing file attributes, location, and permissions, which is crucial for achieving cross-platform file consistency management. Specific metadata content includes, but is not limited to: 1. Path information: the logical path of the file (e.g., \Server\Share\File.txt under the first type of operating system or / mnt / nfs / file.txt under the second type of operating system). 2. Attribute information: filename, file size, creation time, modification time, and access time. Note the differences in timestamp precision and format across different systems (e.g., FILETIME for Type 1 operating systems versus Unix Timestamp for Type 2 operating systems). 3. Permission Information: This is a challenge for cross-platform transmission. Type 1 operating systems use ACLs (Access Control Lists) and SIDs, while Type 2 operating systems use POSIX permission bits (rwx) and UID / GID. The protocol conversion module is responsible for mapping the source's permission model to a model acceptable to the target (e.g., mapping the Type 1 operating system's ACL to the Type 2 operating system's Owner / Group / Other permissions, or preserving extended attributes). 4. Other Attributes: File type (regular file, directory, symbolic link, etc.), special flags (read-only, hidden, system file, etc.).
[0062] Standardized operation instructions are a set of generic instructions generated by the protocol conversion module and understood by the storage intermediary layer. They shield the differences in source protocols and provide a consistent operational semantics for the storage intermediary layer. The instruction mapping engine constructs a structured instruction object based on the extracted file data and metadata, combined with internally defined instruction set specifications. This object no longer contains SMB or NFS-specific protocol headers, but instead includes standardized operation type codes, target resource identifiers, operation parameters, and data payloads. Instruction content can be exemplified as follows: 1. Operation type: uniformly uses internal enumeration values such as OP_WRITE, OP_READ, and OP_METADATA_UPDATE, instead of SMB2_WRITE for SMB or WRITE3 for NFS. 2. Resource identifier: uses a unified logical path or globally unique file ID, instead of protocol-specific File Handle or File Descriptor. 3. Metadata mapping: converts source-side permission bits into permission descriptors common to the intermediary layer. For example, mapping "Full Control" for the first type of operating system to PERMISSION_FULL defined by the intermediary layer. 4. Data encapsulation: The fragmented file data is encapsulated into a standard list of data blocks, each carrying a sequence number and a checksum.
[0063] The above steps overcome protocol barriers, completely eliminating the direct communication obstacles between Type I operating systems (SMB) and Type II operating systems (NFS / POSIX) caused by differences in protocol structure and semantic definitions. It also ensures metadata consistency; through refined metadata extraction and mapping, it ensures that file attributes (especially permissions and timestamps) are not lost or corrupted during cross-platform transmission, solving the metadata distortion problem caused by protocol conversion in existing technologies. Furthermore, it unifies operational semantics, transforming complex and variable network file system protocol requests into single, stable internal instructions. This provides a unified operational foundation for subsequent intelligent routing, load balancing, and heterogeneous storage adaptation in the storage intermediary layer, thereby achieving truly "protocol-independent" storage access.
[0064] Optionally, in the cross-platform file transfer method provided in this embodiment of the invention, the step of generating standardized operation instructions based on file data and metadata includes: parsing source file attribute information from metadata, wherein the source file attribute information includes: permission identifier and creation time information of the source operating system; converting the source file attribute information into target file attribute information, wherein the target file attribute information includes: a target permission model compatible with the target storage system and a unified format timestamp; parsing the target operation type of the file operation request; and encapsulating the file data, target file attribute information, and target operation type into standardized operation instructions.
[0065] The specific implementation process of generating standardized operation instructions based on file data and metadata in this embodiment of the invention first involves deep parsing and cross-platform adaptation of source file attribute information. Specifically, the protocol conversion module extracts metadata from the received file operation request, focusing on parsing the permission identifier and creation time information of the source operating system. Since different operating systems have fundamentally different definitions of file attributes—for example, the first type of operating system typically uses access control lists (ACLs) based on security identifiers (SIDs) to define permissions, while the second type of operating system uses a POSIX permission model based on user IDs (UIDs) and group IDs (GIDs)—and their timestamp formats and precision also differ (the first type of operating system commonly uses the FILETIME structure, while the second type commonly uses Unix timestamps), the core of the above implementation lies in converting the source file attribute information into the target file attribute information. This conversion process includes two key sub-steps: First, the mapping of permission models, which involves converting the permission identifiers on the source end into a target permission model compatible with the target storage system. For example, mapping the ACL permissions of the first type of operating system to the rwx permission bits of the second type of operating system or retaining them as extended attributes for later restoration; Second, the unification of time formats, which involves converting the timestamps on the source end into timestamps of a unified format to ensure the accuracy and consistency of time information when transmitted between heterogeneous systems and to avoid time parsing errors caused by format incompatibility.
[0066] Secondly, the protocol conversion module parses the target operation type in the file operation request. This operation type indicates the specific action the client wants to perform, such as reading, writing, deleting, or modifying attributes, and standardizes it into internally recognizable operation command codes. Finally, the protocol conversion module encapsulates the processed target file data, the converted target file attribute information, and the standardized target operation type into standardized operation commands. These standardized operation commands no longer contain header information or status fields specific to the source protocol, but are presented in a structured data format, facilitating unified parsing and processing by the storage intermediary layer.
[0067] In another optional embodiment, when performing permission mapping, if the source permission identifier cannot be directly mapped to a specific value in the target permission model (such as certain special first-class operating system permission bits), the protocol conversion module may adopt a default mapping strategy, such as granting the target file default read-only or read-write permissions, or retaining the original permission information as extended metadata stored in the target storage system so that it can be restored when needed.
[0068] In addition, after generating standardized operation instructions, the protocol conversion module can dynamically adjust the data fragment size (default value is 4MB) according to network conditions and file size, and divide the file data into multiple data fragments, each with a checksum to ensure data integrity during transmission.
[0069] In terms of metadata management, a two-level caching mechanism (local memory cache and distributed Redis cache) can be used to accelerate path queries and metadata access. When the cache is hit, the metadata in the cache can be returned directly; otherwise, a query request is sent to the storage intermediary layer.
[0070] Regarding data consistency assurance, a two-phase commit protocol (2PC) can be used during write operations. This involves first sending standardized operation instructions to the target storage adapter for pre-commit, and then formally committing the data after confirming that the storage resources are available and the metadata mapping table has been successfully updated. If either phase fails, a rollback operation is performed to ensure data consistency.
[0071] The above implementation enables fine-grained metadata consistency management and an efficient data fragmentation transmission mechanism in cross-platform file transfer. This invention's system ensures 100% metadata consistency after file transfer between heterogeneous systems through precise permission mapping and unified timestamps, resolving attribute loss or corruption issues caused by protocol differences in existing technologies. By dynamically adjusting data fragment sizes and introducing checksum mechanisms, it improves the efficiency and reliability of large-scale file transfers, reduces the impact of network fluctuations on the transmission process, and thus enhances the overall system's transmission performance and user experience.
[0072] Step S203: The standardized operation instructions are sent to the storage intermediary layer, which queries the metadata mapping table to determine the physical storage path and storage type of the target storage system, and selects the target storage adapter based on the storage type.
[0073] It should be noted that the storage intermediary layer is the core coordination component in this embodiment of the invention. It acts as a bridge between the protocol conversion module and the underlying heterogeneous storage cluster. It does not directly store user data, but is responsible for logical routing, resource scheduling, and metadata management. Specifically, routing decision-making involves receiving standardized operation instructions from the protocol conversion module, querying the metadata mapping table based on the file logical path in the instructions, and determining the final physical location of the data. Load balancing refers to the built-in load balancing algorithm that dynamically selects the most suitable storage node to avoid single-point overload and ensure high availability of the storage cluster. Resource abstraction refers to shielding the differences between underlying storage systems (such as HDFS, object storage, and block devices) from the upper layers, providing a unified file operation interface to the lower layers. The communication mechanism involves the storage intermediary layer and the protocol conversion module communicating using gRPC to ensure efficient service calls; the transmission process uses TLS 1.3 encryption to ensure the security of data interaction.
[0074] The metadata mapping table is a core data structure maintained by the storage intermediary layer. It records the correspondence between file logical paths and physical storage locations, and is crucial for heterogeneous storage addressing. This data structure includes the following components: 1. Logical path: The unified namespace path used by clients when accessing files, such as / shared / docs / report.pdf. This is transparent to upper-layer applications and unaffected by the underlying physical storage structure. 2. Physical storage path: The actual storage address of the file in the underlying specific storage system (such as a specific HDFS DataNode, an object storage Bucket, or a block device volume). 3. Storage type identifier: Indicates the type of media on which the file is currently stored (such as HDFS, S3 object storage, local block storage, etc.) to select the correct access protocol. 4. Version information: Supports file version control, records the latest version number of the file, and is used to handle conflict detection during concurrent writes.
[0075] To improve query efficiency, a two-level caching mechanism combining local memory caching and distributed Redis caching is employed. The metadata cache TTL (Time To Live) is set to 60 seconds by default to balance data freshness and query latency. To ensure consistency, version number conflict detection is used to implement multi-client concurrency control, ensuring metadata consistency in high-concurrency scenarios.
[0076] Another point to clarify is that the physical storage path refers to the exact storage location of the file within the underlying heterogeneous storage cluster. Due to the diversity of underlying storage systems, the path formats vary. For example, in HDFS it might be hdfs: / / cluster1 / data / files / report.pdf; in object storage it might be s3: / / bucket-name / path / report.pdf; and in block devices it might be / dev / sdb1 / mnt / data / report.pdf. The storage intermediary layer resolves the logical path into these specific physical paths through a metadata mapping table.
[0077] Storage type refers to the classification of the underlying storage media's technical characteristics. For example: HDFS (Distributed File System): suitable for storing large files and high-throughput scenarios, featuring a data block replication mechanism. Object storage: suitable for storing unstructured data (e.g., images, videos), accessed via HTTP / HTTPS interfaces, and supports unlimited expansion. Block devices: suitable for scenarios requiring low latency and high IOPS, typically manifested as disk volumes. NAS / file storage: traditional network-attached storage, supporting direct access via NFS or SMB protocols.
[0078] The storage intermediary layer dynamically selects the optimal storage backend based on the file type (such as text, video, etc.) and the storage characteristics of the target operating system (such as file system format and permission model). For example, large video files may preferentially choose object storage, while small files may choose HDFS or block storage to optimize small file access performance.
[0079] The target storage adapter is the driver module that enables interaction between the storage intermediary layer and the underlying storage system. It is responsible for converting standardized operation commands into native commands that the specific storage system can recognize and execute. Its functional characteristics include: 1. Protocol Conversion: Converting standardized commands from the intermediary layer into specific protocol commands of the target storage system. For example, if the target storage is HDFS, the adapter converts write commands into HDFS API calls; if it is object storage, it converts them into PUT Object requests. 2. Attribute Mapping Execution: When performing data writes or reads, the adapter handles the specific settings of file attributes. For example, setting the replica count when writing to HDFS, and setting ACLs or metadata headers when writing to object storage. 3. Dynamic Loading: In actual deployment, different storage adapters can be dynamically loaded to extend compatibility, meaning the system can support new storage types through a plug-in approach without modifying the core code. The storage intermediary layer selects the corresponding target storage adapter based on the "storage type" recorded in the metadata mapping table. If the mapping table indicates that the file is stored in object storage, the object storage adapter is loaded; if it indicates that it is stored in HDFS, the HDFS adapter is loaded.
[0080] The above implementation decouples the logical view of files from their physical storage view. Users or clients no longer need to know whether files are actually stored in HDFS, object storage, or block devices; they can access them through a unified logical path, improving system flexibility and maintainability. The combination of metadata mapping tables and storage adapters also enables unified management and intelligent scheduling of various heterogeneous storage resources. The system can automatically distribute data to the most suitable storage medium based on file type, size, and performance requirements, fully leveraging the performance advantages of different storage media. A two-level caching mechanism accelerates metadata queries and reduces frequent network round trips to the storage intermediary layer, improving file access response speed. Simultaneously, dynamically selecting the optimal storage node and adapter ensures load balancing and system stability under high concurrency.
[0081] Optionally, in the cross-platform file transfer method provided in this embodiment of the invention, the step of querying the metadata mapping table by the storage intermediary layer to determine the physical storage path and storage type of the target storage system, and selecting the target storage adapter based on the storage type, includes: extracting the file type identifier from the metadata; querying the metadata mapping table based on the file type identifier to obtain at least one candidate storage node corresponding to the file type identifier, thereby obtaining a list of candidate storage nodes; calculating the load weight of each candidate storage node, and selecting the target storage node from the list of candidate storage nodes based on the load weight; and loading the target storage adapter corresponding to the target storage node according to the storage type.
[0082] In this embodiment of the invention, the specific implementation process of the storage intermediary layer querying the metadata mapping table and selecting the target storage node first extracts a file type identifier from the metadata of standardized operation instructions. This identifier is used to distinguish whether the file is text, video, image, or other specific format, so as to match the most suitable storage backend for files with different characteristics. Subsequently, the protocol conversion module uses the file type identifier as the query key to access the metadata mapping table in the storage intermediary layer. The metadata mapping table not only records the correspondence between the logical path and the physical path of the file, but also maintains the health status, remaining capacity, and supported file type list of each storage node. Through querying, the system obtains at least one candidate storage node that matches the file type identifier, forming a candidate storage node list. These candidate nodes are usually nodes with corresponding storage capabilities (such as object storage interface, HDFS interface, or block device interface) and currently in a normal state.
[0083] Next, the storage intermediary layer performs load balancing decisions. For each candidate storage node in the list, its load weight is calculated dynamically based on the node's current workload (such as CPU utilization, memory utilization, disk I / O throughput, or network bandwidth utilization). Typically, nodes with lower loads are selected with higher weights, or a weighted round-robin algorithm is used to distribute requests. Based on the calculated load weights, an optimal target storage node is selected from the list of candidate storage nodes to ensure overall load balancing of the storage cluster, avoid single-point overload, and thus improve system throughput and response speed.
[0084] After determining the target storage node, the storage intermediary layer dynamically loads or invokes the corresponding target storage adapter based on the storage type represented by that node (e.g., HDFS if the node belongs to an HDFS cluster, and object storage if it belongs to an object storage gateway). The target storage adapter is the driver module connecting the storage intermediary layer and the underlying specific storage system, responsible for converting standardized operation commands into native commands that can be recognized by the specific storage system (such as HDFS API, S3 API, or local file system calls). For example, if the target storage type is HDFS, the HDFS adapter is loaded to convert write commands into HDFS write operations; if it is object storage, the object storage adapter is loaded to convert it into a PUT Object request.
[0085] When querying the metadata mapping table, if a two-level caching mechanism (local memory + distributed Redis) is used, the system first checks whether the metadata mapping information for the file exists in the cache. If the cache is hit, the physical storage path and storage type in the cache are returned directly without accessing the backend database, thus reducing query latency. If the cache is missed, the backend distributed metadata database is queried, and the query result is written to the cache, with a default TTL (Time To Live) of 60 seconds to ensure data freshness.
[0086] The above implementation enables efficient load balancing and intelligent routing in heterogeneous storage clusters. Through a dynamic selection mechanism based on file type and node load weight, the system can fully leverage the performance advantages of different storage media (e.g., HDFS is suitable for large files, and object storage is suitable for massive amounts of unstructured data), avoiding resource hotspots and improving the overall throughput and availability of the storage cluster. Simultaneously, by dynamically loading target storage adapters, transparency and scalability of the storage backend are achieved, allowing the system to seamlessly integrate new storage types without modifying the core routing logic, thus meeting the needs of dynamic changes and heterogeneous coexistence of storage resources in cloud environments.
[0087] Optionally, in the cross-platform file transfer method provided in this embodiment of the invention, the step of determining the physical storage path and storage type of the target storage system by querying the metadata mapping table by the storage intermediary layer further includes: querying the metadata mapping records in the local cache to obtain a first query result; if the first query result is empty, querying the metadata mapping records in the distributed metadata database to obtain a second query result; writing the metadata mapping records queried in the first query result or the second query result into the local cache and setting the cache expiration time; and determining the physical storage path and the corresponding storage type based on all the queried metadata mapping records.
[0088] This embodiment of the invention employs a two-level caching architecture to optimize query performance by querying metadata mapping records to obtain physical storage paths and storage types. First, upon receiving standardized operation instructions, the storage intermediary layer prioritizes querying the metadata mapping cache in its local memory. This local cache stores the mapping relationship between recently accessed file paths and their physical storage locations. If a corresponding record is found in the cache (i.e., the first query result is not empty), this result is directly used to determine the physical storage path and storage type, thereby avoiding network I / O overhead and significantly improving read speed. If the first query result is empty (i.e., a cache miss occurs), the storage intermediary layer initiates a query request to the backend distributed metadata database to obtain the metadata mapping record containing the logical path of the file (i.e., the second query result). The distributed metadata database, as persistent storage, ensures eventual consistency and integrity of the data.
[0089] After obtaining the second query result, the storage intermediary layer writes the record to its local cache and sets a default cache expiration time (TTL), typically 60 seconds. This mechanism aims to balance data freshness and query efficiency: a shorter TTL ensures that when a file migrates between heterogeneous storage systems or its attributes change, the cache expires quickly and the latest information is retrieved from the metadata database, avoiding the reading of outdated path mappings; while a longer TTL reduces the access pressure on the distributed metadata database. Finally, based on the queried metadata mapping record (whether from the local cache or the distributed metadata database), the storage intermediary layer resolves the file's specific physical storage path in the underlying heterogeneous storage system (such as the block location of an HDFS DataNode or the Bucket path of object storage) and the storage type of that path (such as HDFS, object storage, or block device).
[0090] The above implementation achieves a balance between high performance and data consistency in metadata queries. By introducing a local memory caching mechanism, metadata query latency is reduced, improving the response speed of cross-platform file transfers, especially in high-concurrency scenarios, effectively alleviating the pressure on the distributed metadata database. Simultaneously, by setting reasonable cache expiration times and combining them with a two-phase commit protocol and version number conflict detection, metadata consistency and the atomicity of file operations are ensured during cache expiration. This guarantees high performance while avoiding data inconsistencies or access errors caused by outdated caches or concurrency conflicts, improving the overall system reliability and user experience.
[0091] Step S204: Through the target storage adapter, perform cross-platform transfer operations on the file data in the standardized operation instructions and the target storage system based on the physical storage path, obtain the execution results, and send them back to the client.
[0092] Performing cross-platform transfer operations refers to the process by which the target storage adapter uses the loaded underlying storage protocol to actually write or read file data from standardized operation instructions into the physical storage medium. This process is not just a simple data transfer; it also involves the final storage of metadata and status verification.
[0093] During write operations, for large files, the file data is divided into fragments of a default 4MB size. After receiving these fragments, the target storage adapter can perform write operations in parallel, depending on the characteristics of the underlying storage system (such as HDFS's block replication mechanism or object storage's multipart upload mechanism), to improve throughput. Simultaneously with data writing, the adapter writes the previously converted "target file attribute information" (such as a uniformly formatted timestamp and mapped permission bits) to the target storage system's metadata area. For example, when writing to HDFS on a second-tier operating system, corresponding user group permissions and modification timestamps are set. If the two-phase commit protocol is effective at this stage, the adapter first sends a "pre-commit" instruction to the storage system, which reserves space and locks resources; after receiving confirmation, the adapter sends a "final commit" instruction to complete data persistence. If the pre-commit fails, a rollback is triggered, and no actual data is written.
[0094] During read operations, if the target file is stored in a distributed system (such as HDFS or an object storage cluster), the target storage adapter can retrieve file data fragments from multiple underlying storage nodes in parallel, based on the physical storage path. The adapter receives data fragments from different nodes and reassembles them according to the file's logical order, ensuring the integrity and correct sequence of data blocks. During the read process, the adapter compares the checksums of the data fragments to ensure that the data has not been corrupted or lost in the underlying storage.
[0095] The execution result refers to the status information and feedback data returned by the target storage adapter to the storage intermediary layer after completing the cross-platform transfer operation. It includes at least status codes and error information: 1. Success status: Indicates that file data has been successfully written or read, typically including confirmation information such as the size of the file written and the timestamp of creation / update. 2. Failure status: If an error occurs during the transfer (such as insufficient disk space, permission denied, network timeout), the adapter will return a specific error code (such as NFS's NFSERR_ACCES or SMB's error code) and a detailed error description. It may also include: Metadata update results: 1. Version update: If the write is successful, the execution result may include the new version number of the file in the target storage system for subsequent consistency checks and concurrency control. 2. Path confirmation: For newly created files, the execution result confirms its actual path in the physical storage system to update the metadata mapping table. It may also further include performance metrics: In some monitoring scenarios, the execution result may implicitly include performance metrics such as transfer time and throughput for system-level load monitoring and dynamic adjustment.
[0096] Feedback to the client refers to the process by which the storage intermediary layer and protocol conversion module reverse-engineer the execution result back to the client that initiated the request, completing the entire cross-platform interaction loop. This includes the following steps:
[0097] Response message encapsulation: 1. Protocol adaptation: The protocol conversion module receives the execution result from the storage intermediary layer and encapsulates it into the source protocol response message expected by the client. For example, if the client is a Type 1 operating system SMB client, the protocol conversion module encapsulates the successful result into an SMB WRITE response message (including byte count, file handle, etc.); if the client is a Type 2 operating system NFS client, it encapsulates it into an NFS WRITE3 response message. 2. Metadata conversion: If the read operation is successful, the protocol conversion module also needs to convert the metadata in the target storage system (such as POSIX permissions and Unix timestamps of Type 2 operating systems) back to a format that the source client can recognize (such as ACLs and FILETIME timestamps of Type 1 operating systems) and return it along with the file data.
[0098] Data transmission: 1. Data stream return: For read requests, file data and its corresponding metadata are transmitted back to the client via the network through the protocol conversion module. 2. Confirmation mechanism: After receiving the response message, the client parses it according to the protocol specifications. If it is a successful response, the client updates its local cache or completes the file saving operation; if it is an error response, the client may trigger a retry mechanism (maximum of 3 retries) or display an error message to the user.
[0099] Session persistence: The entire feedback process must maintain the consistency of the session state between the client and the protocol conversion module, ensuring that context information such as file handles and session IDs are correctly associated, and avoiding connection interruption due to state loss.
[0100] The above steps enable transparent processing across the entire process, from client-initiated requests to final result retrieval. The client doesn't need to be aware of the underlying HDFS, object storage, or block device, nor does it need to concern itself with protocol conversion details; it only needs to interact with the familiar source protocol to complete cross-platform file operations. Two-phase commit, verification and comparison, and retry mechanisms ensure the atomicity and integrity of file data transmission in complex network environments and heterogeneous storage systems, preventing data loss or corruption. Parallel data fragmentation and reassembly techniques maximize network bandwidth and storage I / O performance, significantly improving the transmission speed of large files in cross-platform environments and solving the problem of low efficiency in traditional single-point serial transmission.
[0101] Optionally, in the cross-platform file transfer method provided in this embodiment of the invention, the step of performing cross-platform transfer operations on file data in a standardized operation instruction and a target storage system based on a physical storage path includes: when the target operation indicated by the standardized operation instruction is a write operation, dividing the file data into at least one data fragment; sending at least one data fragment in parallel to different storage nodes of the target storage system for storage based on the physical storage path; generating a write success confirmation result after receiving storage success signals for all data fragments; and triggering a rollback operation and generating a write failure confirmation result if storage success signals for all data fragments are not received within a preset time window.
[0102] The specific implementation process of performing cross-platform transmission operations on file data in standardized operation instructions based on physical storage paths in this embodiment of the invention first involves the fragmentation of large file data. When the target operation indicated by the standardized operation instruction is a write operation, the target storage adapter will divide the file data to be transmitted into at least one data fragment. The size of the data fragment can be dynamically adjusted according to network conditions, with a typical value of 4MB. This parameter aims to balance transmission efficiency and parallelism. Each fragment carries a unique sequence number and checksum to ensure the correct order and data integrity during subsequent reassembly.
[0103] Subsequently, the target storage adapter, based on the physical storage path, sends at least one data shard in parallel to different storage nodes of the target storage system for storage. Since the target storage system may be a distributed cluster (such as HDFS or object storage cluster), the physical storage path may correspond to multiple actual storage nodes. The adapter utilizes parallel data channels to simultaneously send the corresponding data shards to these nodes, thereby fully utilizing the cluster's bandwidth and I / O capabilities and significantly improving write throughput. During the transmission process, the target storage adapter also creates or updates corresponding metadata records in the underlying storage system, such as file replication policies and block location information.
[0104] Next, the target storage adapter waits to receive storage success signals for all data shards. This synchronization step ensures that all shards have been successfully persisted to the target storage node. According to the data consistency guarantee mechanism, if a two-phase commit protocol (2PC) is used, this phase may correspond to the "formal commit" stage, i.e., confirming that all resources are ready and locked. If storage success signals for all data shards are received within the preset time window, the target storage adapter will generate a write success confirmation result, which includes the written file size, updated timestamp, and other information, and return it to the storage intermediary layer. The preset time window is used to prevent indefinite waiting due to network latency or node failure, ensuring system responsiveness.
[0105] If not all data shards' storage success signals are received within the preset time window, or if only some shard failure signals are received, the target storage adapter will trigger a rollback operation. The rollback operation aims to undo partially written data, ensuring file system consistency and atomicity. For example, it may delete incompletely written shards or mark the file write operation as failed. Subsequently, the adapter generates a write failure acknowledgment, including the specific error reason (such as timeout, node failure, insufficient space, etc.), and feeds it back to the storage intermediary layer, where the protocol conversion module translates it into a protocol error code understandable by the client.
[0106] The above implementation enables efficient, reliable, and atomic transmission of large files between heterogeneous storage systems. By employing a data sharding parallel transmission mechanism, the performance of the distributed storage cluster is maximized, significantly improving the write speed of large files. A two-phase commit protocol and rollback mechanism ensure the atomicity and data consistency of write operations, avoiding data inconsistencies caused by partial writes. Checksum comparison and retry mechanisms enhance the reliability and fault tolerance of data transmission, especially in scenarios with network fluctuations or node failures, ensuring the integrity and availability of file data, thereby improving the overall quality and user experience of cross-platform file transfer.
[0107] Optionally, in the cross-platform file transfer method provided in this embodiment of the invention, the step of performing cross-platform transfer operations on file data in standardized operation instructions and target storage systems based on physical storage paths further includes: when the target operation indicated by the standardized operation instructions is a read operation, determining at least one data fragment location in the target storage system according to the physical storage path; acquiring at least one data fragment in parallel from at least one data fragment location through a parallel data channel; performing integrity verification on the acquired at least one data fragment, and reassembling the at least one data fragment into complete file data after the integrity verification passes.
[0108] This invention embodiment describes the specific implementation process of performing cross-platform transmission operations on file data in standardized operation instructions based on physical storage paths. When the target operation indicated by the standardized operation instruction is a read operation, the target storage adapter first resolves the specific location of at least one data fragment storing the file data in the target storage system according to the previously determined physical storage path. Since the underlying storage system (such as a distributed file system or object storage) typically divides large files into multiple blocks or fragments, the physical storage path not only contains the file identifier but also implicitly includes the distribution information of these fragments within the cluster. The target storage adapter uses this information to simultaneously initiate read requests from different data fragment locations through parallel data channels, thereby acquiring at least one data fragment. This parallel acquisition mechanism leverages the high bandwidth and multi-node concurrent processing capabilities of the distributed storage system, significantly improving the throughput of large file reads.
[0109] After acquiring at least one data fragment, the target storage adapter performs integrity checks on each fragment. Each data fragment carries a checksum during transmission. Upon receiving a fragment, the adapter recalculates the checksum and compares it to the expected checksum carried in the fragment header. If the checksum passes, the fragment is considered intact in the storage medium and undamaged during transmission. If the checksum fails, an appropriate error handling mechanism is triggered, such as retrying the read or marking the fragment as invalid. Only after all acquired data fragments have passed integrity checks will the target storage adapter proceed to the next step: reassembling at least one data fragment into a complete file based on its logical order within the original file. This reassembly process involves not only simple data concatenation but may also involve byte order adjustments or encoding format conversions to ensure the reassembled file is completely identical to the source file.
[0110] The above implementation enables efficient and reliable reading of large files across heterogeneous storage systems. By acquiring data fragments through parallel data channels, the concurrent reading capabilities of the underlying distributed storage cluster are maximized, significantly reducing the reading time for large files and improving user experience. An integrity verification mechanism ensures that the read data is completely consistent with the data in the storage medium, preventing data corruption due to storage medium failure or transmission errors and guaranteeing data reliability. Through data fragment reassembly, the file data received by the client is ensured to be logically complete and ordered, maintaining the correctness of the file semantics. In summary, this solution aims to address the performance bottlenecks and data integrity issues of reading large files in cross-platform, heterogeneous storage environments, providing a high-performance, highly reliable file reading solution.
[0111] Optionally, in the cross-platform file transfer method provided in this embodiment of the invention, the step of obtaining the execution result and feeding it back to the client includes: encrypting the execution result; and sending the encrypted execution result to the client through a secure transmission channel, wherein the secure transmission channel is established using a transport layer security protocol.
[0112] The specific implementation process of obtaining execution results and feeding them back to the client in this embodiment of the invention first involves the data security protection of the execution results. After the storage intermediary layer or protocol conversion module generates the execution results (including file data, metadata, or status confirmation information), the system encrypts these data. The transmission process uses TLS 1.3 (Transport Layer Security version 1.3) for encryption, meaning that the encryption process is not manually encapsulated at the application layer, but is automatically completed by the established secure channel. The protocol conversion module or storage intermediary layer hands the data to be sent to the network stack, and the TLS 1.3 protocol stack is responsible for data encryption, integrity protection (through MAC or AEAD mechanisms), and key exchange. This encryption method ensures that even if the data is eavesdropped on during network transmission, attackers cannot decipher the file content or metadata, thereby protecting the confidentiality of the data.
[0113] Subsequently, the encrypted execution result is sent to the client via a secure transmission channel. This secure transmission channel is established based on the TLS 1.3 protocol. Before data transmission begins, the protocol conversion module and the client perform a TLS handshake process to negotiate cipher suites, exchange public keys, and generate session keys. After the handshake is completed, the communication channel between the two parties is secure. Execution results (whether it's a successful acknowledgment message, a read file data stream, or a failure error code) are all transmitted through this established encrypted channel. Thanks to TLS 1.3, the handshake process is more efficient, reducing round-trip latency, and providing stronger security features than earlier versions, such as Forward Secrecy (PFS), ensuring that even if the private key is leaked long-term, past communication records cannot be decrypted.
[0114] The above implementation achieves data security and privacy protection during cross-platform file transfer. By introducing a TLS 1.3 secure transmission channel, sensitive file data, metadata (such as permissions and paths), and control commands are prevented from being stolen, tampered with, or replayed in untrusted network environments. The efficient handshake and low latency of TLS 1.3 minimize the impact of encryption operations on transmission performance while ensuring high security, guaranteeing high availability and smoothness in cross-platform file transfer. Simultaneously, the end-to-end encryption mechanism meets the stringent requirements of modern enterprise applications for data compliance and security, resolving the security vulnerabilities caused by plaintext data transmission in traditional file-sharing solutions and enhancing the overall security level of the system.
[0115] Through steps S201 to S204 above, file operation requests from clients with heterogeneous operating systems can be received first. These file operation requests are generated based on the source file system protocol. The protocol conversion module then parses the file operation requests, extracts the file data and metadata, and generates standardized operation instructions based on the file data and metadata. These standardized operation instructions are then sent to the storage intermediary layer, which queries the metadata mapping table to determine the physical storage path and storage type of the target storage system. Based on the storage type, it selects the target storage adapter. Finally, through the target storage adapter, cross-platform transfer operations are performed on the file data in the standardized operation instructions and the target storage system based on the physical storage path. The execution results are then obtained and fed back to the client.
[0116] In this embodiment of the invention, a "protocol decoupling and unified abstraction" approach is adopted. Through a dual processing method involving the collaborative work of a "protocol conversion module" and a "storage intermediary layer," the protocol conversion module first parses file operation requests generated based on the source file system protocol, extracts file data and metadata, and generates standardized operation instructions, thus shielding the heterogeneous differences in protocol formats between different operating systems. Subsequently, the storage intermediary layer queries the metadata mapping table based on the metadata to dynamically determine the physical storage path and storage type of the target storage system, and accurately matches the corresponding target storage adapter, achieving flexible mapping between logical instructions and physical storage resources. This process achieves the goal of converting cross-platform file transfer requests into a unified internal format and performing intelligent routing, thereby realizing complete decoupling between the client and the underlying heterogeneous storage system. It eliminates compatibility barriers caused by differences in protocol versions, authentication mechanisms, and file attribute definitions, and thus solves the technical problem of poor protocol compatibility in cross-platform file transfer caused by the lack of a unified heterogeneous storage intermediary coordination mechanism in related technologies.
[0117] The invention will now be described in conjunction with another alternative embodiment.
[0118] Example 2
[0119] This invention also provides a cross-platform file transfer device. It should be noted that the cross-platform file transfer device of this invention includes multiple implementation units, which can be used to execute the cross-platform file transfer method provided in the first embodiment above. Each implementation unit corresponds to each implementation step in the first embodiment above.
[0120] Figure 3 This is a schematic diagram of an optional cross-platform file transfer device according to an embodiment of the present invention, such as... Figure 3 As shown, the device may include: a receiving unit 31, a parsing unit 32, a sending unit 33, and a feedback unit 34.
[0121] The receiving unit 31 is used to receive file operation requests from clients with heterogeneous operating systems, wherein the file operation requests are generated based on the source file system protocol.
[0122] The parsing unit 32 is used to parse the file operation request through the protocol conversion module, extract the file data and metadata in the file operation request, and generate standardized operation instructions based on the file data and metadata.
[0123] The sending unit 33 is used to send standardized operation instructions to the storage intermediary layer, which then queries the metadata mapping table to determine the physical storage path and storage type of the target storage system, and selects the target storage adapter based on the storage type.
[0124] Feedback unit 34 is used to perform cross-platform transfer operations on file data in standardized operation instructions and target storage system based on physical storage path through target storage adapter, obtain execution results and feed them back to client.
[0125] The aforementioned cross-platform file transfer device can first receive file operation requests from clients with heterogeneous operating systems via receiving unit 31. These file operation requests are generated based on the source file system protocol. Then, parsing unit 32 parses the file operation requests via a protocol conversion module, extracts the file data and metadata from the file operation requests, and generates standardized operation instructions based on the file data and metadata. Next, sending unit 33 sends the standardized operation instructions to the storage intermediary layer, which queries the metadata mapping table to determine the physical storage path and storage type of the target storage system. Based on the storage type, it selects the target storage adapter. Finally, feedback unit 34 performs cross-platform transfer operations on the file data in the standardized operation instructions and the target storage system via the target storage adapter, based on the physical storage path, obtains the execution results, and feeds them back to the client.
[0126] In this embodiment of the invention, a "protocol decoupling and unified abstraction" approach is adopted. Through a dual processing method involving the collaborative work of a "protocol conversion module" and a "storage intermediary layer," the protocol conversion module first parses file operation requests generated based on the source file system protocol, extracts file data and metadata, and generates standardized operation instructions, thus shielding the heterogeneous differences in protocol formats between different operating systems. Subsequently, the storage intermediary layer queries the metadata mapping table based on the metadata to dynamically determine the physical storage path and storage type of the target storage system, and accurately matches the corresponding target storage adapter, achieving flexible mapping between logical instructions and physical storage resources. This process achieves the goal of converting cross-platform file transfer requests into a unified internal format and performing intelligent routing, thereby realizing complete decoupling between the client and the underlying heterogeneous storage system. It eliminates compatibility barriers caused by differences in protocol versions, authentication mechanisms, and file attribute definitions, and thus solves the technical problem of poor protocol compatibility in cross-platform file transfer caused by the lack of a unified heterogeneous storage intermediary coordination mechanism in related technologies.
[0127] Furthermore, the parsing unit includes: a first parsing module, used to parse source file attribute information from metadata, wherein the source file attribute information includes: permission identifier and creation time information of the source operating system; a conversion module, used to convert the source file attribute information into target file attribute information, wherein the target file attribute information includes: a target permission model compatible with the target storage system and a unified format timestamp; a second parsing module, used to parse the target operation type of the file operation request; and an encapsulation module, used to encapsulate the file data, target file attribute information, and target operation type into standardized operation instructions.
[0128] Furthermore, the sending unit includes: an extraction module for extracting file type identifiers from metadata; a first acquisition module for querying a metadata mapping table based on the file type identifier to obtain at least one candidate storage node corresponding to the file type identifier, thereby obtaining a list of candidate storage nodes; a calculation module for calculating the load weight of each candidate storage node and selecting a target storage node from the list of candidate storage nodes based on the load weight; and a loading module for loading the target storage adapter corresponding to the target storage node according to the storage type.
[0129] Furthermore, the sending unit also includes: a first query module, used to query metadata mapping records in the local cache to obtain a first query result; a second query module, used to query metadata mapping records in the distributed metadata database to obtain a second query result if the first query result is empty; a writing module, used to write the metadata mapping records queried in the first query result or the second query result into the local cache and set the cache expiration time; and a first determination module, used to determine the physical storage path and the corresponding storage type based on all the queried metadata mapping records.
[0130] Furthermore, the feedback unit includes: a partitioning module, used to partition file data into at least one data fragment when the target operation indicated by the standardized operation instruction is a write operation; a storage module, used to send at least one data fragment in parallel to different storage nodes of the target storage system for storage based on the physical storage path; a generation module, used to generate a write success confirmation result after receiving storage success signals for all data fragments; and a triggering module, used to trigger a rollback operation and generate a write failure confirmation result if storage success signals for all data fragments are not received within a preset time window.
[0131] Furthermore, the feedback unit also includes: a second determining module, used to determine at least one data fragment location in the target storage system according to the physical storage path when the target operation indicated by the standardized operation instruction is a read operation; a second acquiring module, used to acquire at least one data fragment in parallel from at least one data fragment location through a parallel data channel; and a reassembly module, used to perform integrity verification on the acquired at least one data fragment, and reassemble the at least one data fragment into complete file data after the integrity verification passes.
[0132] Furthermore, the feedback unit also includes: an encryption module for encrypting the execution result; and a sending module for sending the encrypted execution result to the client through a secure transmission channel, wherein the secure transmission channel is established using a transport layer security protocol.
[0133] It should be noted that the receiving unit 31, parsing unit 32, sending unit 33, and feedback unit 34 mentioned above correspond to steps S201 to S204 in Embodiment 1. The instances and application scenarios implemented by the above units and corresponding steps are the same, but are not limited to the content disclosed in Embodiment 1. It should be noted that the above modules or units can be hardware or software components stored in memory (e.g., memory 104) and processed by one or more processors (e.g., processors 102a, 102b, ..., 102n). The above modules or units can also be part of a device and run in the computer terminal 10 provided in Embodiment 1.
[0134] The invention will now be described in conjunction with another alternative embodiment.
[0135] Example 3
[0136] The present invention can also provide an electronic device. Figure 4 This is a structural block diagram of an electronic device that performs a cross-platform file transfer method according to an embodiment of the present invention, such as... Figure 4 As shown, the electronic device may include: one or more ( Figure 4(Only one is shown) Processor 402, memory 404, memory controller, and peripheral interface, wherein the peripheral interface is connected to the radio frequency module, audio module and display.
[0137] The memory can be used to store software programs and modules, such as the program instructions / modules corresponding to the cross-platform file transfer method and apparatus in this application embodiment. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, thereby realizing the aforementioned cross-platform file transfer method. The memory may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include memory remotely located relative to the processor, and these remote memories can be connected to the terminal via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.
[0138] The processor can access information and applications stored in memory via a transmission device to perform the following steps: receiving file operation requests from clients with heterogeneous operating systems, wherein the file operation requests are generated based on the source file system protocol; parsing the file operation requests through a protocol conversion module, extracting file data and metadata from the file operation requests, and generating standardized operation instructions based on the file data and metadata; sending the standardized operation instructions to a storage intermediary layer, where the storage intermediary layer queries a metadata mapping table to determine the physical storage path and storage type of the target storage system, and selects a target storage adapter based on the storage type; and performing cross-platform transfer operations on the file data in the standardized operation instructions and the target storage system through the target storage adapter based on the physical storage path, obtaining the execution results and feeding them back to the client.
[0139] The processor can also invoke information and applications stored in the memory via the transmission device to perform the following steps: parsing source file attribute information from metadata, wherein the source file attribute information includes: permission identifiers and creation time information of the source operating system; converting the source file attribute information into target file attribute information, wherein the target file attribute information includes: a target permission model compatible with the target storage system and a timestamp in a unified format; parsing the target operation type of the file operation request; and encapsulating the file data, target file attribute information, and target operation type into standardized operation instructions.
[0140] The processor can also invoke information and applications stored in the memory via the transmission device to perform the following steps: extracting file type identifiers from the metadata; querying the metadata mapping table based on the file type identifiers to obtain at least one candidate storage node corresponding to the file type identifier, thus obtaining a list of candidate storage nodes; calculating the load weight of each candidate storage node, and selecting a target storage node from the list of candidate storage nodes based on the load weights; and loading the corresponding target storage adapter of the target storage node according to the storage type.
[0141] The processor can also invoke information and applications stored in the memory via the transmission device to perform the following steps: query the metadata mapping record in the local cache to obtain a first query result; if the first query result is empty, query the metadata mapping record in the distributed metadata database to obtain a second query result; write the metadata mapping record queried in the first query result or the second query result into the local cache and set the cache expiration time; determine the physical storage path and the corresponding storage type based on all the queried metadata mapping records.
[0142] The processor can also invoke information and applications stored in the memory via the transmission device to perform the following steps: when the target operation indicated by the standardized operation instruction is a write operation, divide the file data into at least one data fragment; send at least one data fragment in parallel to different storage nodes of the target storage system for storage based on the physical storage path; after receiving storage success signals for all data fragments, generate a write success confirmation result; if storage success signals for all data fragments are not received within a preset time window, trigger a rollback operation and generate a write failure confirmation result.
[0143] The processor can also invoke information and applications stored in memory via a transmission device to perform the following steps: when the target operation indicated by the standardized operation instruction is a read operation, determine at least one data fragment location in the target storage system according to the physical storage path; acquire at least one data fragment in parallel from at least one data fragment location via a parallel data channel; perform integrity verification on the acquired at least one data fragment, and after the integrity verification passes, reassemble the at least one data fragment into complete file data.
[0144] The processor can also access information and applications stored in memory via a transmission device to perform the following steps: encrypting the execution result; and sending the encrypted execution result to the client via a secure transmission channel, wherein the secure transmission channel is established using a transport layer security protocol.
[0145] This invention provides a cross-platform file transfer solution. Employing a "protocol decoupling and unified abstraction" approach, it utilizes a dual processing method involving a "protocol conversion module" and a "storage intermediary layer." First, the protocol conversion module parses file operation requests generated based on the source file system protocol, extracts file data and metadata, and generates standardized operation instructions, thus masking the heterogeneous differences in protocol formats between different operating systems. Subsequently, the storage intermediary layer queries a metadata mapping table based on the metadata to dynamically determine the physical storage path and storage type of the target storage system, and accurately matches the corresponding target storage adapter, achieving flexible mapping between logical instructions and physical storage resources. This process achieves the goal of transforming cross-platform file transfer requests into a unified internal format and performing intelligent routing, thereby achieving complete decoupling between the client and the underlying heterogeneous storage system. It eliminates compatibility barriers caused by differences in protocol versions, authentication mechanisms, and file attribute definitions, and solves the technical problem of poor protocol compatibility in cross-platform file transfer caused by the lack of a unified heterogeneous storage intermediary coordination mechanism in related technologies.
[0146] Those skilled in the art will understand that Figure 4 The structure shown is for illustrative purposes only. Electronic devices can also be smartphones, tablets, handheld computers, mobile internet devices (MIDs), PADs, and other terminal devices. Figure 4 This does not limit the structure of the aforementioned electronic device. For example, electronic devices may also include components that are more... Figure 4 The more or fewer components shown (such as network interfaces, display devices, etc.), or having the same Figure 4 The different configurations shown.
[0147] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing the hardware related to the terminal device. The program can be stored in a computer-readable storage medium, which may include: flash drive, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.
[0148] The invention will now be described in conjunction with another alternative embodiment.
[0149] Example 4
[0150] This invention also provides a computer-readable storage medium. Optionally, in this invention, the computer-readable storage medium can be used to store the program code executed by the cross-platform file transfer method provided in Embodiment 1.
[0151] Optionally, in this embodiment of the invention, the storage medium may be located in any computer terminal in a group of computer terminals in a computer network, or in any mobile terminal in a group of mobile terminals.
[0152] This invention also provides a computer program product, which, when executed on a data processing device, is suitable for performing the steps of a cross-platform file transfer method: receiving a file operation request from a client with a heterogeneous operating system, wherein the file operation request is generated based on the source file system protocol; parsing the file operation request through a protocol conversion module, extracting file data and metadata from the file operation request, and generating standardized operation instructions based on the file data and metadata; sending the standardized operation instructions to a storage intermediary layer, whereby the storage intermediary layer queries a metadata mapping table to determine the physical storage path and storage type of the target storage system, and selects a target storage adapter based on the storage type; and performing a cross-platform transfer operation on the file data in the standardized operation instructions and the target storage system through the target storage adapter based on the physical storage path, obtaining the execution result and feeding it back to the client.
[0153] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.
[0154] In the above embodiments of this application, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.
[0155] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling, direct coupling, or communication connection may be through some interfaces; the indirect coupling or communication connection between units or modules may be electrical or other forms.
[0156] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.
[0157] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.
[0158] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard drive, magnetic disk, or optical disk.
[0159] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.
Claims
1. A cross-platform file transfer method, characterized in that, include: Receive file operation requests from clients with heterogeneous operating systems, wherein the file operation requests are generated based on the source file system protocol; The protocol conversion module parses the file operation request, extracts the file data and metadata from the file operation request, and generates standardized operation instructions based on the file data and metadata. The standardized operation instructions are sent to the storage intermediary layer, which queries the metadata mapping table to determine the physical storage path and storage type of the target storage system, and selects the target storage adapter based on the storage type. The target storage adapter performs cross-platform transfer operations on the file data in the standardized operation instructions and the target storage system based on the physical storage path, obtains the execution results, and feeds them back to the client.
2. The cross-platform file transfer method according to claim 1, characterized in that, The steps for generating standardized operation instructions based on the file data and the metadata include: The source file attribute information is parsed from the metadata, wherein the source file attribute information includes: the permission identifier and creation time information of the source operating system; The source file attribute information is converted into target file attribute information, wherein the target file attribute information includes: a target permission model compatible with the target storage system and a timestamp in a unified format; Parse the target operation type of the file operation request; The file data, the target file attribute information, and the target operation type are encapsulated into the standardized operation instructions.
3. The cross-platform file transfer method according to claim 1, characterized in that, The steps of querying the metadata mapping table by the storage intermediary layer to determine the physical storage path and storage type of the target storage system, and selecting the target storage adapter based on the storage type, include: Extract the file type identifier from the metadata; Based on the file type identifier, query the metadata mapping table to obtain at least one candidate storage node corresponding to the file type identifier, and obtain a candidate storage node list. Calculate the load weight of each candidate storage node, and select a target storage node from the list of candidate storage nodes based on the load weight; Load the target storage adapter corresponding to the target storage node according to the storage type.
4. The cross-platform file transfer method according to claim 1, characterized in that, The step of determining the physical storage path and storage type of the target storage system by querying the metadata mapping table from the storage intermediary layer further includes: Query the metadata mapping records in the local cache to obtain the first query result; If the first query result is empty, query the metadata mapping records in the distributed metadata database to obtain the second query result; Write the metadata mapping record found in the first query result or the second query result into the local cache, and set the cache expiration time; The physical storage path and the corresponding storage type are determined based on all the metadata mapping records retrieved.
5. The cross-platform file transfer method according to claim 1, characterized in that, The steps of performing cross-platform transfer operations on the file data in the standardized operation instructions and the target storage system based on the physical storage path include: When the target operation indicated by the standardized operation instruction is a write operation, the file data is divided into at least one data fragment; Based on the physical storage path, the at least one data fragment is sent in parallel to different storage nodes of the target storage system for storage; After receiving storage success signals for all data fragments, a write success confirmation result is generated; If no storage success signal is received for all data shards within the preset time window, a rollback operation is triggered and a write failure confirmation result is generated.
6. The cross-platform file transfer method according to claim 1, characterized in that, The steps of performing cross-platform transfer operations on the file data in the standardized operation instructions and the target storage system based on the physical storage path further include: When the target operation indicated by the standardized operation instruction is a read operation, at least one data fragment location in the target storage system is determined according to the physical storage path; At least one data fragment is acquired in parallel from the at least one data fragment location via a parallel data channel; The integrity of the at least one data fragment is verified, and after the integrity verification is passed, the at least one data fragment is reassembled into complete file data.
7. The cross-platform file transfer method according to claim 1, characterized in that, The steps for obtaining the execution result and sending it back to the client include: The execution result is encrypted; The encrypted execution result is sent to the client through a secure transmission channel, which is established using a transport layer security protocol.
8. A cross-platform file transfer device, characterized in that, include: A receiving unit is configured to receive file operation requests from clients operating heterogeneous operating systems, wherein the file operation requests are generated based on the source file system protocol; The parsing unit is used to parse the file operation request through the protocol conversion module, extract the file data and metadata in the file operation request, and generate standardized operation instructions based on the file data and the metadata; The sending unit is used to send the standardized operation instructions to the storage intermediary layer, which queries the metadata mapping table to determine the physical storage path and storage type of the target storage system, and selects the target storage adapter based on the storage type. The feedback unit is used to perform cross-platform transmission operations on the file data in the standardized operation instructions and the target storage system based on the physical storage path through the target storage adapter, obtain the execution results, and feed them back to the client.
9. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored computer program, wherein, when the computer program is executed, it controls the device on which the computer-readable storage medium is located to perform the cross-platform file transfer method according to any one of claims 1 to 7.
10. An electronic device, characterized in that, It includes one or more processors and a memory, the memory being used to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors cause the one or more processors to implement the cross-platform file transfer method according to any one of claims 1 to 7.
11. A computer program product, characterized in that, Includes computer instructions, wherein when executed by a processor, the computer instructions implement the steps of the cross-platform file transfer method according to any one of claims 1 to 7.
