Method, device and computer equipment for accessing an object storage platform

Abstract

The application relates to a method and device for accessing an object storage platform and computer equipment, and relates to the technical field of cloud computing storage. The method in one embodiment comprises the following steps: receiving a data synchronization instruction and responding to the data synchronization instruction, uploading data under a first data directory to a target storage space of a target object storage platform; mounting the target storage space of the target object storage platform to a second data directory locally of an application server; performing a renaming operation on the first data directory, and creating a redirection mechanism for redirecting operations on the first data directory to the second data directory. The scheme of the embodiment can realize non-intrusive access to the object storage platform, and reduces the cost of accessing the object storage platform.

Description

Methods, apparatus and computer equipment for accessing object storage platforms Technical Field

[0001] This application relates to the field of cloud computing storage technology, and in particular to a method for accessing an object storage platform, an apparatus for accessing an object storage platform, a computer device, a computer-readable storage medium, and a computer program product. Background Technology

[0002] As cloud computing becomes the infrastructure of the digital economy, Object Storage Service (OSS) has become the de facto standard for storing unstructured data (such as documents, images, videos, and backup archives) due to its advantages such as massive capacity, high reliability, elastic scalability, and pay-as-you-go pricing. Enterprises generally have a strong need to migrate data from existing applications from local file systems or traditional network-attached storage to cloud-based object storage platforms that provide object storage services, thereby reducing IT (Internet Technology) operation and maintenance costs, improving data reliability, and leveraging the ecosystem capabilities of cloud platforms.

[0003] Against this backdrop, enabling applications to access object storage efficiently, securely, and conveniently has become a key technical challenge. To access object storage platforms, applications typically rely on native RESTful APIs (Web APIs conforming to the REST (Representational State Transfer) architectural style) provided by cloud service providers for programmatic access. This approach is extremely flexible but highly complex to develop. To address this, cloud service providers have launched pre-packaged Software Development Kits (SDKs), simplifying API (Application Programming Interface) calls and becoming the mainstream choice for new application development. However, this access method is an intrusive integration approach, forcing applications to make fundamental changes at the code level to adapt to cloud storage. This constitutes a significant technical barrier for applications that cannot or are unable to modify their code to access cloud storage services. Summary of the Invention

[0004] Therefore, it is necessary to provide a method, apparatus, computer device, computer-readable storage medium, and computer program product that can access an object storage platform without modifying application code and without intrusion, in order to address the above-mentioned technical problems.

[0005] Firstly, this application provides a method for accessing an object storage platform, the method being applied to an application server, the method comprising:

[0006] Receive and respond to data synchronization instructions, and upload the data in the first data directory to the target storage space of the target object storage platform;

[0007] Mount the target storage space of the target object storage platform to the second data directory on the local application server;

[0008] Perform a renaming operation on the first data directory and create a redirection mechanism to redirect operations on the first data directory to the second data directory.

[0009] In some embodiments, a redirection mechanism is created to redirect operations on a first data directory to a second data directory, including:

[0010] Create a symbolic link with the same name as the first data directory, and point the symbolic link to the second data directory.

[0011] In some embodiments, performing a renaming operation on the first data directory includes:

[0012] Rename the first data directory to the backup data directory.

[0013] In some embodiments, the method further includes:

[0014] Receive file operation requests sent by the kernel processor of the application server. The file operation request is sent by the kernel processor after receiving a file operation request from the application and the file operation request includes a first data directory, by relocating the file operation request to a second data directory.

[0015] Intercept file operation requests and convert them into network requests;

[0016] Send the network request to the target object storage platform.

[0017] In some embodiments, the method further includes:

[0018] Receive file data returned by the target object storage platform based on a network request;

[0019] The file data is sent back to the kernel processor, and then sent back to the application via the kernel processor.

[0020] In some embodiments, the method further includes:

[0021] When the conditions for regular monitoring are met, a status query command is sent to the second data directory;

[0022] If no status query response is received for the status query command within the predetermined time period, the connection with the target object storage platform will be re-established.

[0023] In some embodiments, before receiving and responding to a data synchronization instruction, the method further includes: stopping the application's application service;

[0024] After creating a redirection mechanism that redirects operations on the first data directory to the second data directory, the method also includes: restarting the application.

[0025] In some embodiments, receiving and responding to a data synchronization instruction, uploading data from a first data directory to a target storage space of a target object storage platform includes:

[0026] Receive and respond to the data synchronization command, and upload all data in the first data directory to the target storage space of the target object storage platform.

[0027] In some embodiments, receiving and responding to a data synchronization instruction, uploading data from a first data directory to a target storage space of a target object storage platform includes:

[0028] Receive and respond to data synchronization instructions, and determine the first data directory for incremental data access from multiple data directories;

[0029] Upload all data in the first data directory to the target storage space of the target object storage platform.

[0030] Secondly, this application also provides an apparatus for accessing an object storage platform, the apparatus being configured on an application server, the apparatus comprising:

[0031] The data synchronization module is used to receive and respond to data synchronization instructions, and upload the data in the first data directory to the target storage space of the target object storage platform;

[0032] The mounting module is used to mount the target storage space of the target object storage platform to the second data directory on the local application server.

[0033] The redirection processing module is used to perform a renaming operation on the first data directory and create a redirection mechanism that redirects operations on the first data directory to the second data directory.

[0034] Thirdly, this application also provides a computer device. The computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the steps of the method for accessing the object storage platform in any of the above embodiments.

[0035] Fourthly, this application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program thereon, which, when executed by a processor, implements the steps of the method for accessing the object storage platform in any of the above embodiments.

[0036] Fifthly, this application also provides a computer program product. The computer program product includes a computer program that, when executed by a processor, implements the steps of the method for accessing the object storage platform in any of the above embodiments.

[0037] The aforementioned method, apparatus, computer equipment, storage medium, and computer program product for accessing an object storage platform synchronize the application server's data to the data storage platform by uploading data from the first data directory of the application server to the target storage space of the target object storage platform. After uploading the data from the first data directory to the target object storage platform, the target storage space of the target object storage platform is mounted to a second data directory on the application server's local machine. Furthermore, after renaming the first data directory, a redirection mechanism is created to redirect operations on the first data directory to the second data directory, thereby synchronizing the application server's local data... In essence, after the data in the original first data directory is synchronized to the object storage platform and mounted locally on the application server, a redirection mechanism is established to redirect operations from the first data directory to the second data directory. This allows the application server to access the object storage platform and provide application services when it subsequently makes requests involving the first data directory, based on the redirection mechanism and the mounting mechanism of the second data directory where the target storage space is mounted locally on the application server. During this process, the data deployed to the object storage platform is uploaded from the application server without requiring any changes or adjustments to the software code. Therefore, it achieves non-intrusive access to the object storage platform without modifying the application code, which can greatly improve the development efficiency of adding the object storage platform and reduce development costs. Attached Figure Description

[0038] Figure 1 is a schematic diagram of an application scenario of a method for accessing an object storage platform in one embodiment;

[0039] Figure 2 is a flowchart illustrating a method for accessing an object storage platform in one embodiment;

[0040] Figure 3 is a flowchart illustrating a method for accessing an object storage platform in another embodiment;

[0041] Figure 4 is a flowchart illustrating the method for accessing an object storage platform in another embodiment;

[0042] Figure 5 is a schematic diagram of the structure of an access object storage platform in one embodiment;

[0043] Figure 6 is a schematic diagram of the deployment architecture of a device accessing an object storage platform in a specific example;

[0044] Figure 7 is a flowchart illustrating a specific example of a method for accessing an object storage platform;

[0045] Figure 8 is a schematic diagram of the structure of a computer device in one embodiment. Detailed Implementation

[0046] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0047] The embodiments of the technical solution of this application will now be described in detail with reference to the accompanying drawings. These embodiments are only used to more clearly illustrate the technical solution of this application and are therefore merely examples, and should not be used to limit the scope of protection of this application.

[0048] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains; the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the application; the terms “comprising” and “having”, and any variations thereof, in the specification, claims, and foregoing description of the drawings are intended to cover non-exclusive inclusion.

[0049] In the description of the embodiments of this application, technical terms such as "first" and "second" are used only to distinguish different objects and should not be construed as indicating or implying relative importance or implicitly specifying the number, specific order, or primary and secondary relationship of the indicated technical features. In the description of the embodiments of this application, "multiple" means two or more, unless otherwise explicitly defined.

[0050] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0051] In the description of the embodiments in this application, the term "and / or" is merely a description of the relationship between associated objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this document generally indicates that the preceding and following associated objects have an "or" relationship, and the term "multiple" refers to two or more (including two).

[0052] It should be noted that all information and data involved in this application (including but not limited to data used for analysis, stored data, and displayed data) are information and data authorized by the user or fully authorized by all parties, and the acquisition, transmission, storage, use, and processing of the relevant data comply with the relevant provisions of national laws and regulations. In the embodiments of this application, certain existing industry solutions such as software, components, and models may be mentioned. These should be considered exemplary, and their purpose is merely to illustrate the feasibility of implementing the technical solution of this application, but does not imply that the applicant has already used or necessarily used such a solution.

[0053] Currently, to access object storage platforms, applications typically can only programmatically access them through the native RESTful APIs provided by cloud service providers. This approach is extremely flexible but also highly complex to develop. To address this, cloud service providers have launched pre-packaged Software Development Kits (SDKs), simplifying API (Application Programming Interface) calls and becoming the mainstream choice for new application development.

[0054] While SDK-based integration offers straightforward functionality, it exposes a series of significant technical problems and limitations when facing complex real-world enterprise IT environments, especially when migrating existing systems to the cloud or building hybrid cloud architectures:

[0055] On the one hand, the source code of legacy systems or third-party commercial software may be unavailable, unmodifiable, or structurally complex and difficult to understand. If SDK integration is adopted, a radical code-level overhaul of the application is necessary, requiring significant investment in development, testing, and verification resources. This is time-consuming, costly, and highly susceptible to introducing new errors, impacting system stability. Clearly, such applications are highly invasive and the cost of modification is exorbitant.

[0056] On the other hand, once an application's code is deeply integrated with a particular cloud service provider's SDK, switching cloud service providers in the future would almost require repeating the same level of code modification work. This results in enterprises being "tied" to a single cloud vendor, losing flexibility and bargaining power in technology selection, leading to high technology lock-in risk and poor migration flexibility.

[0057] On the other hand, enterprises may have a large number of applications that use standard file system interfaces (such as POSIX (Portable Operating System Interface) standards), which access files through simple system calls such as open, read, write, and close. These programs may be written in different languages ​​(such as C / C++, Java, Python, PHP) and have different architectures. Using an SDK solution means that it is necessary to adapt it separately for each language and each application, which is a huge workload and difficult to manage in a unified manner. Therefore, there are huge challenges in terms of compatibility and adaptability.

[0058] On the other hand, when using SDK integration, the application needs to directly handle low-level details such as network communication, data segmentation, and retry mechanisms, which increases the complexity of the application itself and its resource consumption (such as CPU and memory). At the same time, the management of authentication keys also brings additional security challenges, requiring careful handling to avoid leakage, which complicates the architecture and increases the difficulty of maintenance.

[0059] In summary, the core problem with SDK integration lies in its "intrusiveness," forcing applications to make fundamental changes at the code level to adapt to cloud storage. This constitutes a significant technical barrier for applications that cannot or are unable to modify their code to access cloud storage services. Therefore, there is an urgent need for a new technological solution that can achieve "non-intrusive" or "transparent" access to object storage services to overcome these shortcomings.

[0060] Research has shown that by using operating system-level file system mounting and symbolic linking technologies, combined with data synchronization tools, seamless and transparent access to cloud storage services on object storage platforms can be achieved without modifying any line of application code. This approach can transform complex cloud storage API access into local file system operations that applications are unaware of, effectively solving the challenges of legacy system transformation, multi-platform migration, and unified data access in hybrid cloud architectures.

[0061] Accordingly, this application provides a method for accessing an object storage platform, which can be applied to the application environment shown in Figure 1. The application server 102 communicates with the object storage platform 104 via a network. The object storage platform 104 is a platform that provides object storage services, offering massive, secure, and low-cost cloud storage services. The application server 102 runs an application that needs to access the object storage platform 104. After the application data from the application server 102 is accessed to the object storage platform 104, data access and access can be achieved through the object storage services provided by the object storage platform 104. The type of application server 102 is not limited; it can be a physical server or a virtual machine, and this application does not impose specific limitations.

[0062] This application provides an apparatus 1022 for accessing an object storage platform, which can execute a method for accessing the object storage platform. The apparatus 1022 can be deployed on an application server 102. It can synchronize the local file system or traditional network attached storage of the application server, specifically the relevant data of the application server (including but not limited to application-related data), to the object storage platform 104, and establish a storage space for storing the data on the object storage platform 104. This data is then mounted on the application server 102, achieving non-intrusive access to the object storage platform 104. Through the redirection mechanism of the directory of the application server's local data to the mount point, it is beneficial to use the server provided by the object storage platform 104 for data application when providing application services. Thus, while achieving non-intrusive access to the object storage platform 104, the application server can also use the object storage service provided by the object storage platform 104 to perform application operations.

[0063] Accordingly, in one embodiment, as shown in FIG2, a method for accessing an object storage platform is provided, which is applied to an application server 102 and can be executed by a device 1022 on the application server for accessing the object storage platform, including the following steps:

[0064] Step S201: Receive and respond to the data synchronization command, and upload the data in the first data directory to the target storage space of the target object storage platform.

[0065] A data synchronization command is a command used to trigger the synchronization of data that needs to be migrated from the application server to the object storage platform for cloud object storage. The method of obtaining the data synchronization command is not limited. In some examples, it may be a command triggered by the application server 102, the object storage platform 104, or a third-party terminal or server when an operation to access the object storage platform 104 is required. In other examples, it may be determined that the data synchronization command has been received when a custom or pre-set trigger time point for data synchronization is reached on the application server, or when other custom or pre-set trigger conditions on the application server are met.

[0066] The specific method for triggering the data synchronization command is not limited. When the data synchronization command is triggered based on application server 102, object storage platform 104, or a third-party terminal or server, it can be based on user operation, triggering application server 102, object storage platform 104, or a third-party terminal or server to send the data synchronization command to application server 102. Additionally, custom or pre-set triggering time nodes or triggering conditions for data synchronization can also be set based on actual needs. After receiving the data synchronization command, the device 1022 connected to the object storage platform performs subsequent processing operations.

[0067] The first data directory is the data directory of the application server's local file system or traditional network attached storage that needs to be migrated to the object storage platform and stored in the cloud on the object storage platform. It can be the data directory of an application service provided by the application server, but is not limited to this.

[0068] Understandably, multiple service providers can offer multiple different object storage platforms. The target object storage platform is based on this data synchronization instruction, and the data on the application server that this access operation expects to be migrated to the object storage platform.

[0069] When uploading data from the first data directory to the target object storage platform, the data from the first data directory can be synchronized to the target storage space of the target object storage platform. The method for determining the target storage space is not limited; it can be based on the storage space determined by the mechanism of the target object storage platform, but is not limited to this.

[0070] Step S202: Mount the target storage space of the target object storage platform to the second data directory on the local application server.

[0071] There are no restrictions on how the target storage space is mounted to the second data directory on the local application server. Existing mounting methods in related technologies can be used, and this application does not impose any specific limitations on this method.

[0072] After mounting the target storage space to the second data directory on the application server, the target storage space of the target object storage platform can be read and written by accessing the second data directory.

[0073] The method of mounting the target storage space of the target object storage platform to the second data directory on the application server is not limited. For example, in some examples, it can be implemented based on the FUSE (Filesystem in Userspace) framework to mount the target storage space of the target object storage platform to the second data directory on the application server, but it is not limited to this. During the process of mounting the target storage space of the target object storage platform to the second data directory on the application server, a network connection with the object storage service of the target object storage platform can be established, and authentication information such as AccessKey can be used to complete the authentication, but it is not limited to this.

[0074] Step S203: Perform a renaming operation on the first data directory and create a redirection mechanism to redirect operations on the first data directory to the second data directory.

[0075] By renaming the first data directory, previous requests to that directory will no longer access it. Simultaneously, a redirection mechanism is created to redirect operations from the first data directory to the second data directory. Therefore, when access to the first data directory is needed, since it has been renamed and this redirection mechanism exists, access to the first data directory will be redirected to the second data directory. This allows reading and writing of data in the target storage space of the target object storage platform to be done through the second data directory.

[0076] The method of renaming the first data directory is not limited. In some embodiments, renaming the first data directory includes:

[0077] Rename the first data directory to the backup data directory.

[0078] Based on this embodiment, after uploading the data in the first data directory to the object storage platform, the first data directory is renamed to the backup file directory. On the one hand, when data needs to be accessed later, it is accessed through the object storage platform by mounting and redirection. On the other hand, renaming the original first data directory to the backup data directory makes it easier to find backup text and compare and determine incremental data when incremental synchronization is needed later.

[0079] The method for accessing the object storage platform based on this embodiment synchronizes the application server's data to the data storage platform by uploading data from the first data directory of the application server to the target storage space of the target object storage platform. After uploading the data from the first data directory to the target object storage platform, the target storage space of the target object storage platform is mounted to the second data directory on the application server, thus mounting the target storage space of the target object storage platform to the application server. After renaming the first data directory, a redirection mechanism is created to redirect operations on the first data directory to the second data directory, thereby realizing the synchronization of the application server's local data, i.e., the original data in the first data directory. After the data is synchronized to the object storage platform, it is mounted locally on the application server. A redirection mechanism is established to redirect operations from the first data directory to the second data directory. This allows the application server to access the object storage platform and provide application services when it makes subsequent requests involving the first data directory, based on the redirection mechanism and the mounting mechanism of the second data directory where the target storage space is mounted locally on the application server. During this process, the data deployed to the object storage platform is uploaded from the application server without requiring any changes or adjustments to the software code. This achieves non-intrusive access to the object storage platform without modifying the application code, which can greatly improve the development efficiency of adding the object storage platform and reduce development costs.

[0080] In step S203 above, the method of creating a redirection mechanism that redirects operations on the first data directory to the second data directory is not limited. In some examples, the redirection mechanism can be implemented through hard links or binding mounts, while in some examples, the redirection mechanism can be implemented through symbolic links.

[0081] Accordingly, in some embodiments, a redirection mechanism is created to redirect operations on the first data directory to the second data directory, including:

[0082] Create a symbolic link with the same name as the first data directory, and point the symbolic link to the second data directory.

[0083] A symbolic link is a special file containing a path string pointing to a target file or directory. When a symbolic link is accessed, the system automatically "follows" the path string and applies it to the final target. It is transparent to most programs and can achieve cross-system operation. Therefore, it can be used with application servers and object storage platforms that use different programming languages ​​and different development code.

[0084] Based on this embodiment, when creating a redirection mechanism that redirects operations on the first data directory to the second data directory, a symbolic link with the same name as the first data directory is created, and the symbolic link points to the second data directory. Since the symbolic link itself only stores the target path and hardly occupies any extra data blocks, it is particularly suitable for avoiding duplicate storage when referencing the same large file or directory in multiple places, and can achieve flexible relocation and transfer.

[0085] In some embodiments, referring to FIG3, the above method further includes:

[0086] Step S301: Receive a file operation request sent by the kernel processor of the application server. The file operation request is sent by the kernel processor after receiving a file operation request from the application and the file operation request includes the first data directory, by relocating the file operation request to the second data directory.

[0087] When an application needs to use the application server provided by the application server, it sends a file operation request to the application server. This file operation request requests to read or write data in the relevant directory of the application server (such as the first data directory mentioned above) to complete the application's processing operations. This file operation request from the application server is received by the application server's kernel processor. When the kernel processor receives the file operation request, it accesses and reads the data in the first data directory. Since the original first data directory has been renamed and cannot be read, and the current first data directory has been redirected to the second data directory, the kernel processor accesses and reads the data in the second data directory instead.

[0088] As mentioned above, the second data directory is the target storage space of the target object storage platform, which is mounted to the application server's mount directory. Therefore, after the kernel processor receives the file operation request, the device connected to the object storage platform can also receive the file operation request from the kernel processor.

[0089] Step S302: Intercept the file operation request and convert it into a network request.

[0090] After receiving the file operation request from the kernel processor, the device accessing the object storage platform intercepts the file operation request and converts it into a network request to access the target object storage platform. The second data directory is the target storage space of the target object storage platform and is mounted to the mount directory of the application server.

[0091] It is understood that there are no limitations on how file operation requests are converted into network requests, and the method can be determined based on the communication protocol requirements of the target object storage platform. This application does not impose any specific limitations on this.

[0092] Step S303: Send the network request to the target object storage platform.

[0093] After generating a network request, the network request can be sent to the target object storage platform, thereby enabling access to the data in the target storage space on the target object storage platform.

[0094] Based on this embodiment, when the kernel processor of the application server receives a file operation request, since the first data directory requested by the file operation has been redirected to the second data directory, and the second data directory is the directory where data synchronized to the object storage platform is mounted to the application server, the file operation request is intercepted based on this mounting mechanism. After converting the file operation request into a network request, the network request is sent to the object storage platform, thus realizing data access during file operations without intrusive access to the object storage platform.

[0095] In some embodiments, referring to FIG4, the method further includes:

[0096] Step S401: Receive file data returned by the target object storage platform based on the network request.

[0097] Based on the received network request, the target object storage platform can obtain the corresponding file data in the target storage space requested by the network request and return it to the sender of the network request, that is, to the application server.

[0098] Since the network request is generated by the device accessing the object storage platform, the file data returned by the target object storage platform will ultimately be returned to the device accessing the object storage platform.

[0099] Step S402: The file data is sent back to the kernel processor, and then sent back to the application via the kernel processor.

[0100] After receiving the file data, the device accessing the object storage platform can send the file data back to the kernel processor, which then sends it back to the application, thus achieving the return of the obtained file data along the original path.

[0101] Based on this embodiment, after receiving file data returned by the object storage platform based on a network request, the file data is sent back to the kernel processor, and then sent back to the application via the kernel processor. For the application, the entire process of reading file data is interactive with the kernel processor, which is no different from reading local files, thus achieving seamless access to data from the object storage platform.

[0102] In some embodiments, the method further includes:

[0103] When the conditions for regular monitoring are met, a status query command is sent to the second data directory;

[0104] If no status query response is received for the status query command within the predetermined time period, the connection with the target object storage platform will be re-established.

[0105] Among them, the periodic monitoring conditions refer to the conditions for periodically monitoring the devices connected to the object storage platform, or the conditions for periodically monitoring the mounting status of the target storage space of the target object storage platform to the application server. In the description of the relevant examples, it can also be referred to as the conditions for periodically monitoring the mount point.

[0106] The specific setting of the periodic monitoring conditions is not limited. In some examples, the periodic monitoring conditions may be determined to be met when a preset time period (e.g., 30 seconds) is reached. In other examples, the periodic monitoring conditions may be determined to be met when a preset operation is detected. The preset operation may include, but is not limited to, not limited to, not receiving a file operation request after a preset interval.

[0107] The type of status query command sent to the second data directory is not limited, as long as the response of the second data directory to the status query command can determine the mount status of the target storage space of the target object storage platform to the second data directory of the application server, that is, the mount status of the mount point can be determined. In some examples, the status query command can be a lightweight operation command, such as the ls command, but it is not limited to this, to check whether the second data directory is in a normal response state, but it is not limited to this.

[0108] If no status query response is received within the predetermined time, it indicates that the mounting status of the target storage space of the target object storage platform to the second data directory of the application server is abnormal. Therefore, the connection with the target object storage platform can be re-established to realize the self-healing function of the mounting status.

[0109] There are no restrictions on how the connection to the target object storage platform can be re-established. Some examples may involve automatically executing the umount and mount commands to re-establish the connection to the target object storage platform, but this is not the only option. As long as the connection to the target object storage platform can be re-established, it is acceptable.

[0110] Based on this embodiment, when the periodic monitoring conditions are met, a status query command is sent to the second data directory. If no status query response is received for the status query command within a predetermined time, the connection with the target object storage platform is re-established, thus realizing the inspection and self-healing of the mounting status.

[0111] In some embodiments, the method further includes:

[0112] The validity of symbolic links is checked periodically. If a symbolic link is broken (e.g., the secondary data directory it points to does not exist), the correct symbolic link can be automatically recreated, thus enabling symbolic links to self-heal.

[0113] The method for verifying the validity of symbolic links is not limited and can be any existing method in related technologies. This application does not impose any specific limitations on this method.

[0114] In some embodiments, the method further includes:

[0115] When the target object storage platform is inaccessible for a predetermined period of time, the system switches to degrade mode. In degrade mode, new file read and write operations are redirected to a preset local degraded storage directory, and all file operation logs are recorded during the period when the system is redirected to the preset local degraded storage directory.

[0116] Therefore, in the event that the target object storage platform cannot be accessed for an extended period (up to the predetermined duration), switching to degrade mode and redirecting new file read / write operations to a preset local degraded storage directory can ensure that core business operations are not interrupted. Furthermore, by recording all file operation logs during the period of redirection to the preset local degraded storage directory, it is beneficial to synchronize the data modified during the degrade period with the target object storage platform based on the logs after network recovery, thereby restoring data consistency.

[0117] In some embodiments, before receiving and responding to a data synchronization instruction, the method further includes: stopping the application's application service;

[0118] After creating a redirection mechanism that redirects operations on the first data directory to the second data directory, the method also includes: restarting the application.

[0119] Based on this embodiment, before performing data synchronization operations, the application service of the application is stopped, and after the redirection mechanism that redirects operations on the first data directory to the second data directory, the application is restarted. This can achieve strong consistency between the data on the application server and the data synchronized to the object storage platform, thereby improving the strong consistency of the data synchronized to the object storage platform.

[0120] The method of receiving and responding to the data synchronization command and uploading the data in the first data directory to the target storage space of the target object storage platform is not limited. It can be a full data synchronization or an incremental data synchronization. Examples are given below.

[0121] In some embodiments, receiving and responding to a data synchronization instruction, uploading data from a first data directory to a target storage space of a target object storage platform includes:

[0122] Receive and respond to the data synchronization command, and upload all data in the first data directory to the target storage space of the target object storage platform.

[0123] The full data in the first data directory refers to all data within that directory. Uploading the full data in the first data directory to the target storage space of the target object storage platform allows for a one-time upload of all data from the first data directory, preventing data loss and improving data synchronization accuracy. Typically, when first connecting to the target object storage platform, the full data in the first data directory is uploaded to the target storage space.

[0124] Based on this embodiment, upon receiving a data synchronization instruction, all data in the first data directory is uploaded to the target storage space of the target object storage platform, realizing the synchronization and access of all data, which helps to avoid data omissions and improve the accuracy of data synchronization.

[0125] In some embodiments, receiving and responding to a data synchronization instruction, uploading data from a first data directory to a target storage space of a target object storage platform includes:

[0126] Upon receiving and responding to the data synchronization command, the system uploads all data in the first data directory to the target storage space of the target object storage platform through multiple incremental synchronizations.

[0127] There are no restrictions on the methods for performing multiple incremental synchronizations. In some examples, data synchronization instructions can be determined based on monitored events, and incremental data can be determined accordingly. For example, incremental data can be determined based on an incremental synchronization mechanism. Some examples of incremental synchronization mechanisms may include an inotify+rsync incremental synchronization mechanism, that is, using inotify on the application server to listen for directory events (creation, modification, deletion, movement, attribute changes, etc.), and calling rsync to incrementally synchronize the changes to the target object storage platform after the event is triggered, but it is not limited to this.

[0128] Based on this embodiment, upon receiving a data synchronization instruction, all data in the first data directory is uploaded to the target storage space of the target object storage platform through multiple incremental synchronizations. This allows for multiple incremental data uploads, followed by subsequent mounting, renaming, and redirection processes. This enables hot data migration, ensuring continuous business operation during data upload to the object storage platform and achieving uninterrupted data access.

[0129] In some examples, the incremental data in the first data directory can be the data from all file operation logs recorded during the period when the data was redirected to a preset local degraded storage directory after switching to degraded mode in the examples above.

[0130] In some embodiments, receiving and responding to a data synchronization instruction, uploading data from a first data directory to a target storage space of a target object storage platform includes:

[0131] Receive and respond to data synchronization instructions, and determine the first data directory for incremental data access from multiple data directories;

[0132] Upload all data in the first data directory to the target storage space of the target object storage platform.

[0133] These multiple data directories can be data directories on the application server that need to be connected to the object storage platform, or data directories that are not yet connected to the object storage platform. For example, the existing system on the application server that needs to be connected to the object storage platform includes multiple directories. For example, let's assume a data catalog. If it has already been connected to the object storage platform, then the directory is now... The data in the database has not yet been connected to the object storage platform, so it can be obtained from... Select one or more data directories as the first data directory for incremental data access.

[0134] Therefore, upon receiving a data synchronization command, each data access operation can only access a portion of the data directory, thus achieving hot synchronization based on time or events. It is understandable that for data already accessed in the data object platform's data directory, due to the redirection mechanism, even when the application server is running, the accessed data can access the object storage platform for services through the redirection mechanism and the mounted second data directory. This allows data access to be performed while the system requiring data migration on the application server is running, ensuring continuous business operation during data access and achieving uninterrupted data access.

[0135] There are no restrictions on the method for determining the first data directory for incremental data access from multiple data directories. In some examples, based on the first time point and the second time point, the data directory whose most recent modification time is between the first time point and the second time point can be used as the first data directory for incremental data access.

[0136] There are no restrictions on how the first and second time points are determined. In some examples, the first and second time points are the time points specified when data synchronization is required, that is, the data synchronization instruction can include the first and second time points.

[0137] In other examples, the first time point can be a time point specified when data synchronization is required, and the second time point can be the current time point. That is, the data synchronization instruction can include the first time point. The incremental data between the first time point and the second time point refers to all data from the first time point to the current time.

[0138] In other examples, the first time point can be the time point of the last data synchronization, and the second time point can be the current time point. Therefore, the data synchronization operation can be performed based on the newly generated or modified data directory after the last data synchronization.

[0139] In other examples, data synchronization instructions can be determined based on monitored events, and the first data directory for incremental data access can be determined accordingly. For example, incremental data can be determined based on an incremental synchronization mechanism. Some examples of incremental synchronization mechanisms may include an inotify+rsync incremental synchronization mechanism, that is, using inotify on the application server to listen for directory events (creation, modification, deletion, movement, attribute changes, etc.), and calling rsync to incrementally synchronize the changes to the target object storage platform after the event is triggered, but it is not limited to this.

[0140] Accordingly, this application also provides an apparatus for implementing the above-described method for accessing an object storage platform. The solution provided by this apparatus is similar to the implementation described in the above-described method. Therefore, the specific limitations in one or more apparatus embodiments for accessing an object storage platform provided below can be found in the limitations of the method for accessing the object storage platform described above, and will not be repeated here.

[0141] In one embodiment, referring to FIG5, an apparatus for accessing an object storage platform is provided, comprising: a data synchronization module 501, a mounting processing module 502, and a redirection processing module 503, wherein:

[0142] The data synchronization module 501 is used to receive and respond to data synchronization instructions, and upload the data in the first data directory to the target storage space of the target object storage platform.

[0143] The mounting processing module 502 is used to mount the target storage space of the target object storage platform to the second data directory on the local application server.

[0144] The redirection processing module 503 is used to perform a renaming operation on the first data directory and create a redirection mechanism that redirects operations on the first data directory to the second data directory.

[0145] In some embodiments, the redirection processing module 503 is used to create a symbolic link with the same name as the first data directory, and the symbolic link points to the second data directory.

[0146] In some embodiments, the redirection processing module 503 is used to rename the first data directory to a backup data directory.

[0147] In some embodiments, the redirection processing module 503 is used to receive a file operation request sent by the kernel processor of the application server. The file operation request is sent by redirecting the file operation request to a second data directory when the kernel processor receives a file operation request from an application and the file operation request includes a first data directory.

[0148] The mounting processing module 502 is used to intercept file operation requests, convert file operation requests into network requests, and send the network requests to the target object storage platform.

[0149] In some embodiments, the mounting processing module 502 is used to receive file data returned by the target object storage platform based on a network request;

[0150] The redirection processing module 503 is used to send file data back to the kernel processor, so that it can be sent back to the application via the kernel processor.

[0151] In some embodiments, the mounting processing module 502 is used to send a status query instruction to the second data directory when the periodic monitoring conditions are met; and to re-establish the connection with the target object storage platform if no status query response is received for the status query instruction within a predetermined period of time.

[0152] In some embodiments, the data synchronization module 501 is configured to stop the application service of the application before receiving and responding to the data synchronization instruction; and to restart the application after the redirection processing module 503 creates a redirection mechanism that redirects operations on the first data directory to the second data directory.

[0153] In some embodiments, the data synchronization module 501 is used to receive and respond to a data synchronization instruction to upload all data in the first data directory to the target storage space of the target object storage platform.

[0154] In some embodiments, the data synchronization module 501 is used to determine a first data directory for incremental data access from multiple data directories; and upload all data in the first data directory to the target storage space of the target object storage platform.

[0155] Based on the example described above, a schematic diagram of the system structure between the application server 102, which is equipped with a device for accessing the object storage platform, and the object storage platform 104 is shown in Figure 6. The process of non-intrusively accessing the data in the application server 102 to the object storage platform 104 can be shown in Figure 7.

[0156] As can be seen from Figures 6 and 7, the scenario involved in the method and apparatus for accessing the object storage platform in this application embodiment mainly includes an application server 102, an object storage platform 104 (providing object storage services), and a network connecting the two. By deploying an apparatus for accessing the object storage platform, which consists of software modules, on the application server 102, the apparatus works in conjunction with the hardware of the application server to redirect the data path, thereby achieving non-intrusive access to the target object storage platform. It can be seen that, in essence, it realizes a non-intrusive access proxy for accessing the object storage platform.

[0157] Referring to Figures 6 and 7, when data needs to be non-intrusively accessed from the application server 102 to the object storage platform 104, the device for accessing the object storage platform (non-intrusive access agent) provided in this embodiment can be deployed and configured on the application server 102. The configured information may include, but is not limited to, the authentication information of the object storage platform 104, the mount point path (e.g., the target storage space of the object storage platform 104, and / or the aforementioned second data directory), and the original data directory path (the aforementioned first data directory). The core hardware of the application server 102 includes a central processing unit (CPU), memory, a local hard disk (such as an HDD / SSD), and a network interface card (NIC). Applications can read and write files to local paths (such as / app / data / ) in a conventional manner through the file system interface of the operating system kernel.

[0158] After deploying and configuring the device to access the object storage platform, application services can be paused, such as applications that read or write to the original data directory (i.e., the first data directory mentioned above), to ensure data consistency.

[0159] The system checks if the application service has stopped. If it hasn't stopped, an alarm is triggered, and it's determined whether the retry count has reached the retry threshold. If the retry threshold has been reached, the process of connecting to the object storage platform ends. If the retry threshold hasn't been reached, the application service is paused during the retry.

[0160] If the application service has stopped, the data synchronization module performs a full data synchronization. For example, the data synchronization module starts the data synchronization engine and uploads all files under the first data directory (e.g., " / app / data / ") to the corresponding target storage space (Bucket) of the target object storage platform for the first time and completely.

[0161] Subsequently, the mount processing module performs the mount operation, mounting the target storage space (Bucket) of the target object storage platform to the specified directory on the application server, namely the second data directory (e.g., " / mnt / oss / ").

[0162] Subsequently, it is determined whether the data synchronization and mounting operations were successfully executed. The methods for determining whether data synchronization and mounting operations were successful can be those already existing in related technologies, and this application embodiment does not limit these methods.

[0163] If either the data synchronization or mounting operation fails, an alarm will be triggered and the system will revert to local backup data, ending the process of connecting to the object storage platform.

[0164] If both data synchronization and mounting operations are successful, the redirection processing module will rename the original data directory (e.g., the first data directory mentioned above) to the backup directory (e.g., " / app / data_backup / "), and immediately create a symbolic link to the first data directory (e.g., " / app / data / ") pointing to the corresponding path under the mount point, i.e., execute the second data directory (e.g., " / mnt / oss / data / ").

[0165] After creating the symbolic link, the data synchronization module can restart the application. When the application needs to access data in the original first data directory, it can access the data in the target object storage platform through the symbolic link, completing a non-intrusive access process.

[0166] Subsequently, as shown in Figure 6, when the application needs to access data in the original first data directory, the application will initiate a file operation request. This file operation request can be a read file request, such as open(" / app / data / file.txt").

[0167] The application server's kernel processor can receive the file operation request, parse the file path of the file operation request, and find that the path of the file operation request (e.g., " / app / data / ") is a symbolic link whose target is the second data directory (e.g., / mnt / oss / data / ). Then the kernel processor automatically redirects the file operation request to the target path, that is, to the second data directory.

[0168] Since the second data directory is the mount point, the kernel processor sends the file operation request to the device accessing the object storage platform, for example, by passing the file operation request to the mount processing module (running in user space) through the kernel processor's VFS (virtual file system).

[0169] After receiving the file operation request, the mounting processing module converts the file operation request into a network request to the target object storage platform, such as a GetObject API request to the target object storage platform, and sends it to the target object storage platform over the network, such as via the application server's network card.

[0170] After receiving the network request, the target object storage platform processes it and returns the corresponding file data to the application server. The application server's network interface card receives this file data and sends it back to the mount processing module.

[0171] After receiving the data, the mount processing module returns it to the kernel processor via VFS, and then passes it to the application via the kernel processor.

[0172] As can be seen from the above, in the entire process of handling file operation requests, the application appears to be accessing a local file. The process of redirection and obtaining data from the target object storage platform is imperceptible to the application, and the whole process is no different from reading a local file.

[0173] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0174] Each module in the aforementioned access object storage platform device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the operations corresponding to each module.

[0175] In an exemplary embodiment, a computer device is provided, which may be a server, and its internal structure diagram is shown in Figure 8. The computer device includes a processor, memory, input / output interfaces (I / O), and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is connected to the system bus via the I / O interfaces. The processor of the computer device provides computing and control capabilities. The memory of the computer device includes non-volatile storage media and internal memory. The non-volatile storage media stores an operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The database of the computer device is used to store data. The I / O interfaces of the computer device are used for exchanging information between the processor and external devices. The communication interface of the computer device is used for communicating with external terminals or servers via a network connection. When the computer program is executed by the processor, it implements a method for accessing an object storage platform.

[0176] Those skilled in the art will understand that the structure of the computer device described above is only a partial structure related to the solution of this application and does not constitute a limitation on the computer device to which the solution of this application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have different component arrangements.

[0177] In some embodiments, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the method for accessing an object storage platform in any of the above embodiments.

[0178] In some embodiments, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps of the method for accessing an object storage platform in any of the above embodiments.

[0179] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps of the method for accessing an object storage platform in any of the above embodiments.

[0180] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0181] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0182] The above embodiments are merely illustrative of several implementation methods of this application, and their descriptions are relatively specific and detailed. However, they should not be construed as limiting the scope of this application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A method for accessing an object storage platform, characterized in that, The method is applied to an apparatus for accessing an object storage platform of an application server. The method includes: receiving and responding to a data synchronization instruction to upload data in a first data directory to a target storage space of a target object storage platform, wherein the first data directory is the data directory of an application provided by the application server; mounting the target storage space of the target object storage platform to a second data directory on the local machine of the application server; performing a renaming operation on the first data directory and creating a redirection mechanism to redirect operations on the first data directory to the second data directory; receiving a file operation request sent by the kernel processor of the application server, wherein the file operation request is sent by redirecting the file operation request to the second data directory when the kernel processor receives a file operation request from an application and the file operation request includes the first data directory, and the file operation request is used to request reading and writing data in a relevant directory of the application server; intercepting the file operation request and converting the file operation request into a network request; sending the network request to the target object storage platform; receiving file data returned by the target object storage platform based on the network request; and transmitting the file data back to the kernel processor for transmission back to the application via the kernel processor.

2. The method according to claim 1, characterized in that, The creation of a redirection mechanism that redirects operations on the first data directory to the second data directory includes: creating a symbolic link with the same name as the first data directory, the symbolic link pointing to the second data directory.

3. The method according to claim 1, characterized in that, The renaming operation on the first data directory includes renaming the first data directory to a backup data directory.

4. The method according to claim 2, characterized in that, The method also includes: periodically verifying the validity of symbolic links, and automatically recreating correct symbolic links if a symbolic link is broken.

5. The method according to claim 1, characterized in that, The method further includes: when the duration of inaccessibility to the target object storage platform reaches a predetermined duration, switching to a degraded mode, in which new file read / write operations are redirected to a preset local degraded storage directory, and all file operation logs during the period of redirection to a preset local degraded storage directory are recorded.

6. The method according to any one of claims 1 to 5, characterized in that, The method further includes: when the periodic monitoring conditions are met, sending a status query instruction to the second data directory; if no status query response is received for the status query instruction within a predetermined period of time, re-establishing the connection with the target object storage platform.

7. The method according to any one of claims 1 to 5, characterized in that, The step of receiving and responding to a data synchronization instruction to upload data in the first data directory to the target storage space of the target object storage platform includes: receiving and responding to a data synchronization instruction to upload all data in the first data directory to the target storage space of the target object storage platform.

8. The method according to any one of claims 1 to 5, characterized in that, The step of receiving and responding to a data synchronization instruction to upload data in the first data directory to the target storage space of the target object storage platform includes: receiving and responding to a data synchronization instruction to determine a first data directory for incremental data access from multiple data directories; and uploading all data in the first data directory to the target storage space of the target object storage platform.

9. An apparatus for accessing an object storage platform, characterized in that, The device is installed on an application server and includes: a data synchronization module for receiving and responding to data synchronization instructions, uploading data in a first data directory to a target storage space of a target object storage platform, wherein the first data directory is the data directory of an application service provided by the application server; a mounting processing module for mounting the target storage space of the target object storage platform to a second data directory on the local machine of the application server; a redirection processing module for performing a renaming operation on the first data directory and creating a redirection mechanism to redirect operations on the first data directory to the second data directory; the redirection processing module is further configured to receive file operation requests sent by the kernel processor of the application server, wherein the file operation request is sent by the kernel processor after receiving a file operation request from an application, and the file operation request includes the first data directory, by redirecting the file operation request to the second data directory, and sending the file data received by the mounting processing module back to the kernel processor for transmission back to the application via the kernel processor; the mounting processing module is further configured to intercept the file operation request, convert the file operation request into a network request, send the network request to the target object storage platform, and receive file data returned by the target object storage platform based on the network request.

10. A computer device comprising a memory and a processor, the memory storing a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 8.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 8.

12. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 8.

Images