Data processing method and device based on object storage system, equipment and medium

Abstract

The invention provides a data processing method and device based on an object storage system, equipment and a medium. In the process of migrating data of a source object storage system to a target object storage system, a first request is intercepted and forwarded; when it is confirmed that the first request exists, the first request of which the type is write data is forwarded to the source object storage system according to the indication of the first configuration information, and when it is confirmed that the second request does not exist, the first configuration information is adjusted to the second configuration information, and the first request of which the type is write data is forwarded to the source object storage system according to the indication of the second configuration information, by comparing the data difference between the source server and the target server and synchronizing the data, the consistency of the data can be ensured and smooth migration can be realized on the premise of not interrupting related services of the source server, so that a user does not sense the data migration process.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a data processing method and device, an electronic device, and a storage medium. Background Art

[0002] Object Storage System (OSS) is a data storage system that stores data as objects rather than in a file hierarchy or block storage method. It is widely used in large-scale data storage needs, such as cloud storage, backup, archiving, and big data analysis.

[0003] Object storage systems usually require data migration. For example, in the initial stage of a business, an existing object storage system is used directly for the purpose of rapid launch. However, as the business scale grows, the original object storage system architecture is no longer applicable, and a new object storage system is redesigned. The data of the original object storage system needs to be migrated to the new object storage system.

[0004] In the related art, when migrating data from an old object storage system to a new object storage system, all services of the old object storage system are stopped to ensure data consistency and data migration stability. However, for some key businesses, stopping services is unacceptable, which will not only affect the normal operation of the front-end business, but also bring a bad experience to users. Summary of the invention

[0005] In view of the above problems, a method, apparatus, device, and medium for data processing based on an object storage system are proposed to overcome the above problems or at least partially solve the above problems, including:

[0006] A method for data processing based on an object storage system, the method comprising:

[0007] In response to a request to migrate data, migrate the data in the source object storage system to the target object storage system;

[0008] In the process of migrating data from a source object storage system to a target object storage system, intercepting a first request; wherein the first request is a service request initiated by a client to the source object storage system;

[0009] Acquiring first configuration information, and forwarding the first request according to the first configuration information, includes: when the type of the first request is to write data, forwarding the first request to the source object storage system according to an instruction of the first configuration information;

[0010] In the process of forwarding the first request according to the first configuration information, querying through the source object storage system whether there is a second request; wherein the second request is a service request received by the source object storage system from the client without being intercepted;

[0011] If the second request does not exist, obtaining second configuration information, and adjusting the first configuration information to the second configuration information;

[0012] Forwarding the first request according to the second configuration information includes: when the type of the first request is to write data, forwarding the first request to the target object storage system according to an instruction of the second configuration information;

[0013] Comparing the difference between the data stored in the source object storage system and the data stored in the target object storage system, determining the difference data, and performing data synchronization on the source object storage system and the target object storage system according to the difference data;

[0014] Provides feedback on the results of the migration data.

[0015] Optionally, forwarding the first request according to the first configuration information further includes:

[0016] In a case where the type of the first request is to read data, determining whether the source object storage system has target data corresponding to the first request according to an instruction of the first configuration;

[0017] If the target data exists, forwarding the first request to the source object storage system;

[0018] If the target data does not exist, forwarding the first request to the target object storage system;

[0019] The forwarding of the first request according to the second configuration information further includes:

[0020] In a case where the type of the first request is to read data, determining whether the target object storage system has target data corresponding to the first request according to an instruction of the second configuration;

[0021] If the target data exists, forwarding the first request to the target object storage system;

[0022] If the target data does not exist, the first request is forwarded to the source object storage system.

[0023] Optionally, forwarding the first request according to the first configuration information further includes:

[0024] In a case where the type of the first request is to upload data in segments, confirming whether the source object storage system has segment data corresponding to the first request according to an instruction of the first configuration;

[0025] If the source object storage system has the shard data corresponding to the first request, forwarding the first request to the source object storage system;

[0026] If the source object storage system does not have the shard data corresponding to the first request, forwarding the first request to the target object storage system;

[0027] The forwarding of the first request according to the second configuration information further includes:

[0028] In the case where the type of the first request is to upload data in pieces, confirming whether the target object storage system has piece data corresponding to the first request according to an instruction of the second configuration;

[0029] If the target object storage system has the shard data corresponding to the first request, forwarding the first request to the target object storage system;

[0030] If the target object storage system does not have the shard data corresponding to the first request, the first request is forwarded to the source object storage system.

[0031] Optionally, forwarding the first request according to the first configuration information further includes:

[0032] When the type of the first request is to check whether a container exists, or to obtain a data list corresponding to a container, forwarding the first request to the source object storage system according to an instruction of the first configuration;

[0033] The forwarding of the first request according to the second configuration information further includes:

[0034] In the case that the type of the first request is to check whether a container exists, or to obtain a data list corresponding to a container, the first request is forwarded to the target object storage system according to an instruction of the second configuration.

[0035] Optionally, forwarding the first request according to the first configuration information further includes:

[0036] When the type of the first request is to operate on a container, forwarding the first request to the target object storage system according to an instruction of the first configuration;

[0037] In a case where the type of the first request is to delete data, forwarding the first request to the source object storage system and the target object storage system according to an instruction of the first configuration;

[0038] The forwarding of the second request according to the second configuration information further includes:

[0039] When the type of the second request is to operate on a container, forwarding the second request to the target object storage system according to an instruction of the second configuration;

[0040] In the case that the type of the second request is to delete data, the second request is forwarded to the source object storage system and the target object storage system according to the instruction of the second configuration.

[0041] Optionally, after synchronizing the source object storage system and the target object storage system according to the difference data, the method further includes:

[0042] Determine whether there is the first request of type reading data;

[0043] If the first request of the type of reading data does not exist, obtaining third configuration information, and adjusting the second configuration information to the third configuration information;

[0044] According to the instruction of the third configuration information, all the first requests are forwarded to the target object storage system.

[0045] Optionally, comparing the difference between the data stored in the source object storage system and the data stored in the target object storage system to determine the difference data includes:

[0046] Traversing each first data in the source object storage system and each second data stored in the target object storage system respectively;

[0047] If the target object storage system does not have the second data corresponding to the first data, determining the first data as the difference data;

[0048] If the target object storage system has the second data corresponding to the first data, verifying whether the first data and the second data are consistent;

[0049] If the first data and the second data are inconsistent, determining the first data as the difference data;

[0050] If the second data is data deleted in the source object storage system, the second data is determined as the difference data.

[0051] A device for data processing based on an object storage system, the device comprising:

[0052] A data migration module, configured to migrate data from a source object storage system to a target object storage system in response to a request to migrate data;

[0053] A first request interception module is used to intercept a first request during the process of migrating data from a source object storage system to a target object storage system; wherein the first request is a service request initiated by a client to the source object storage system;

[0054] A first forwarding module, configured to obtain first configuration information, and forward the first request according to the first configuration information, including: when the type of the first request is to write data, forwarding the first request to the source object storage system according to an instruction of the first configuration information;

[0055] A query module, configured to query whether a second request exists through the source object storage system during the process of forwarding the first request according to the first configuration information; wherein the second request is a service request received by the source object storage system from the client without being intercepted;

[0056] A first configuration adjustment module, configured to obtain second configuration information and adjust the first configuration information to the second configuration information if the second request does not exist;

[0057] A second forwarding module, configured to forward the first request according to the second configuration information, comprising: when the type of the first request is to write data, forwarding the first request to the target object storage system according to an instruction of the second configuration information;

[0058] A data synchronization module, used to compare the difference between the data stored in the source object storage system and the data stored in the target object storage system, determine the difference data, and synchronize the data between the source object storage system and the target object storage system according to the difference data;

[0059] The result feedback module is used to feedback the results of the migration data.

[0060] An electronic device comprises a processor, a memory and a computer program stored in the memory and capable of running on the processor, wherein the computer program implements the data processing method as described above when executed by the processor.

[0061] A computer-readable storage medium, characterized in that a computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, the data processing method as described above is implemented.

[0062] The embodiments of the present invention have the following advantages: the embodiments of the present invention intercept and forward the first request during the process of migrating data from the source object storage system to the target object storage system, forward the first request of the type of writing data to the source object storage system according to the indication of the first configuration information, adjust the first configuration information to the second configuration information when it is confirmed that the second request does not exist, forward the first request of the type of writing data to the source object storage system according to the indication of the second configuration information, compare the data difference between the source server and the destination server and synchronize the data, thereby ensuring data consistency without interrupting the related business of the source server, achieving smooth migration, and making the user unaware of the data migration process. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] In order to more clearly illustrate the technical solution of the present invention, the accompanying drawings required for use in the description of the present invention will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.

[0064] Figure 1 is a flowchart of a method for data processing based on an object storage system provided by an embodiment of the present invention;

[0065] Figure 2 is a schematic diagram of a scanning tool provided by an embodiment of the present invention;

[0066] Figure 3 is a schematic diagram of a synchronization tool provided by an embodiment of the present invention;

[0067] Figure 4 It is an initial flow diagram of a service request provided by an embodiment of the present invention;

[0068] Figure 5 This is a schematic diagram of the flow of a service request after configuration modification provided by an embodiment of the present invention;

[0069] Figure 6 This is another schematic diagram of the flow of service requests after configuration modification provided by an embodiment of the present invention;

[0070] Figure 7 This is another schematic diagram of the flow of service requests after configuration modification provided by an embodiment of the present invention;

[0071] Figure 8 This is a schematic diagram of the flow of a service request after data migration provided by an embodiment of the present invention;

[0072] Fig. 9It is a structural block diagram of a data processing device based on an object storage system provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0073] In order to make the above-mentioned purposes, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0074] With the development of cloud computing, object storage, as a native cloud storage solution, is becoming more and more widely used, and various behaviors on the cloud are almost inseparable from the support of object storage.

[0075] Consider the following scenario: In the initial stage of the business, for the purpose of rapid launch, the existing object storage system was directly used. However, as the business scale grew, the original object storage system architecture was no longer applicable, and a new object storage system was redesigned. The data of the original object storage system needed to be migrated to the new object storage system. This is one of the scenarios for object storage migration. In fact, in the process of using object storage, with the increase in data volume and technological advancement, or considering factors such as cost and data security, enterprises may need to migrate data from one object storage system to another to improve performance, reduce costs, or meet other business needs. At this time, data migration of the object storage system is a very important link.

[0076] For data migration between different object storage systems, many migration tools are provided in related technologies, and some public cloud vendors also provide online migration services.

[0077] The above migration tools or services are basically business-led and perform migration according to the following process:

[0078] 1. Use a bucket (a container for storing objects) as the migration target.

[0079] 2. Stop the service for the storage bucket.

[0080] 3. Traverse the above buckets to get a list of all object names.

[0081] 4. Traverse the above list, read data from the source object storage system, write it to the destination object storage system and perform data verification.

[0082] 5. Re-enable the service using the new access address.

[0083] However, for some businesses, stopping is unacceptable, and coordinating business migration is very troublesome and takes a long time. If the business has data writing operations, serious data inconsistency problems will occur if the business is not stopped. For example, if the business still writes data to the source object storage system after the migration is completed, the business's read and write operations will not be able to read the data when it is switched to the destination object storage system.

[0084] Based on the defects in the above-mentioned related technologies, the present invention proposes a method, device, equipment, and medium for data processing based on an object storage system. In the process of migrating all the data of the source object storage system to the target object storage system, the client's request for the source object storage system is intercepted, and the write request is forwarded to the source object storage system; when it is confirmed that there is no client request for the source object storage system, the write request is forwarded to the target object storage system, and then the difference data between the source object storage system and the target object storage system is synchronized (i.e., incremental migration), thereby realizing smooth data migration and making the user unaware of the data migration process.

[0085] The embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0086] Reference Figure 1 , shows a flowchart of a method for data processing based on an object storage system provided by an embodiment of the present invention, which may specifically include the following steps:

[0087] Step 101, in response to a request to migrate data, migrate data in a source object storage system to a target object storage system;

[0088] In step 101, when a request for data migration is received, data migration is started to migrate all data in the source object storage system to the target object storage system. During this process, the service of the source object storage system is not stopped.

[0089] In some embodiments of the present invention, the present invention further designs a migration agent system, which implements smooth migration of data through three core components, which are specifically as follows:

[0090] 1) Scanning tool: The scanning tool is mainly used to scan data. In the first scan, the scanning tool scans all the data to be migrated in the source object storage system and records the scan results in the migration agent system. Figure 2 As shown, the scanning tool performs a full scan from the source object storage system while performing a full scan from the migration proxy system, and compares and records the scan results, for example, records the difference data.

[0091] In addition, the scanning tool can also be designed to be reentrant, that is, the scanning tool can be safely called by multiple tasks or threads at the same time to ensure the stability of data migration.

[0092] 2) Synchronization tools: such as Figure 3 As shown, the synchronization tool is mainly used to perform a full scan from the migration agent system, and synchronize the data of the source object storage system from the source gateway (i.e., the gateway of the source object storage system) to the target object storage system through the destination gateway (i.e., the gateway of the target object storage system) according to the information recorded by the scanning tool, and record the synchronization results.

[0093] Specifically, the scanning tool can be regarded as an out-of-band operation log recording system, and the synchronization tool performs synchronization according to the operation log. In this way, the scanning tool and the synchronization tool can implement the business logic of full data migration and incremental migration.

[0094] 3) Migration proxy gateway (or migration proxy service): By setting the resolution of bucket domain names in DNS (Domain Name System), the client's business requests for the source object storage system are intercepted, and then different types of business requests are forwarded to the source object storage system and / or the target object storage system. That is, the migration proxy gateway is mainly used to control the flow of business requests.

[0095] The migration proxy gateway can also implement fine-grained business request parsing, parsing each business request into a specific object storage S3 (Simple Storage Service, a communication protocol for object storage) request instead of a simple coarse-grained read and write request.

[0096] In some examples, the request to migrate data may be a request to migrate all objects in a bucket in a source object storage system (hereinafter referred to as a source bucket). In combination with the above migration agent system, the implementation process of fully migrating the data in the source bucket is as follows:

[0097] 1. Determine the source bucket to be migrated. First, synchronize the user information (including access credentials) of the source bucket to the target object storage system. Then synchronize the bucket information to the target object storage system (including operations such as creating a bucket and configuring bucket permission policies).

[0098] 2. Import the information of the source bucket and the destination bucket (i.e., the corresponding bucket in the target object storage system) into the migration agent system, mainly including the bucket domain (to distinguish the source object storage system and the target object storage system), access IP address, and access credentials.

[0099] 3. Configure the proxy record, which contains the bucket name, source domain, and destination domain to be synchronized. Set the initial proxy rule of the migration proxy gateway and name it all_to_src, which means that all requests are forwarded to the source object storage system. Figure 4 shown.

[0100] 4. Use the scanning tool to scan, and then use the synchronization tool to migrate all data.

[0101] Step 102: intercepting a first request during the process of migrating data from the source object storage system to the target object storage system; wherein the first request is a service request initiated by a client to the source object storage system;

[0102] In step 102, some types of business requests can be directly processed in the target object storage system without affecting the consistency of the original business and data, such as reading data, operating on buckets, etc. Therefore, the objects providing services can be gradually transferred from the source object storage system to the target object storage system. Based on this, it is necessary to first uniformly intercept the business requests initiated by the client to the source object storage system, and then forward them to the source object storage system and / or the target object storage system according to different types of business requests to achieve control over the flow of business requests.

[0103] In the specific implementation, the first request can be intercepted by modifying the DNS configuration. For example, modify the DNS resolution configuration of the bucket domain name, and change the destination address of the service request initiated by the client from the source gateway IP address to the IP address of the proxy server (such as the above-mentioned migration proxy system). Since the modification of the DNS settings will not take effect immediately on the client, some service requests still flow directly to the source object storage system through the client at this time, and these service requests are the service requests that have not been intercepted.

[0104] Step 103, obtaining first configuration information, and forwarding the first request according to the first configuration information, including: when the type of the first request is to write data, forwarding the first request to the source object storage system according to the instruction of the first configuration information;

[0105] In step 103, the first configuration information is configuration information indicating the rule for controlling the flow of the service request, ie, the forwarding rule. The first configuration information defines different types of first requests to be forwarded to the source object storage system and / or the target object storage system through different configuration items.

[0106] Since the business request of type write data will affect the data consistency of the source object storage system and the target object storage system after data migration, and in order not to affect the original business, the business request of type write data will still be forwarded to the source object storage system.

[0107] In a specific implementation, the first configuration information may be a proxy rule of the proxy server, and the proxy rule may be named write_to_src, that is, forwarding the data request to the source.

[0108] In some examples, after the initial proxy rule all_to_src of the migration proxy gateway is modified to write_to_src, the flow of the service request is as follows: Figure 5 At this time, there are both service requests (read objects, write objects, etc.) initiated directly by the client to the source object server and service requests forwarded by the migration proxy gateway (whose target address is the IP address of the source gateway and / or the destination gateway).

[0109] Step 104: in the process of forwarding the first request according to the first configuration information, querying through the source object storage system whether there is a second request; wherein the second request is a service request received by the source object storage system from the client without being intercepted;

[0110] In step 104, the second request is a service request directly initiated by the client to the source object storage system without being intercepted and forwarded during the data migration process. Since there is a time difference in the effectiveness of the interception of the first request, some service requests cannot be intercepted. For example, after the DNS configuration is modified, some clients cannot immediately perform domain name resolution according to the modified DNS configuration due to cache settings, etc., then these clients will still directly initiate service requests to access the source storage system without being intercepted and forwarded.

[0111] In a specific implementation, whether the second request exists may be confirmed by querying the access log of the source system.

[0112] As an example, when the migration proxy gateway forwards the first request, it can set the UserAgent in the request header to "s3-sync-proxy", which means that the service request is forwarded by the migration proxy gateway. When querying the access log of the source system, the UserAgent information can be used to distinguish which are the first requests that have been intercepted and forwarded, and which are the second requests that have not been intercepted.

[0113] Step 105: if the second request does not exist, obtain second configuration information, and adjust the first configuration information to the second configuration information;

[0114] In step 105, similar to the first configuration information, the second configuration information is also used to define the forwarding rules for the first request, but the configuration items of the second configuration information and the first configuration information are different. When it is found that the second request does not exist, it can be considered that all client service requests for the source object storage system have been intercepted, and the forwarding rules for the first request need to be adjusted.

[0115] In actual applications, if no newly added second request is found within the preset time, it can also be considered that the second request does not exist, and all client service requests to the source object storage system have been intercepted. For example, during data migration, if the second request is found at time t1, but no newly added second request is found after time t2, it can also be considered that the second request does not exist.

[0116] Step 106, forwarding the first request according to the second configuration information, includes: when the type of the first request is to write data, forwarding the first request to the target object storage system according to the instruction of the second configuration information;

[0117] In step 106, since the second request no longer exists, the data writing service can be completely taken over by the target object storage system. At this time, the data writing service request will not affect the data consistency after the data migration.

[0118] Based on this, the first configuration information is adjusted to the second configuration information, and the first request of the write data type is forwarded to the target object storage system through the second configuration information.

[0119] In a specific implementation, the second configuration information may be a proxy rule of the proxy server, and the proxy rule may be named write_to_dest, that is, the write data request is forwarded to the target object storage system.

[0120] As an example, after the proxy rule write_to_src of the migration proxy gateway is changed to write_to_dest, the flow of the service request is as follows: Figure 6 At this time, only part of the first request (such as a read data request) is forwarded to the source gateway, and all the first requests of the write data type have been forwarded to the destination gateway.

[0121] In some embodiments of the present invention, forwarding the first request according to the first configuration information further includes:

[0122] In a case where the type of the first request is to read data, determining whether the source object storage system has target data corresponding to the first request according to an instruction of the first configuration;

[0123] If the target data exists, forwarding the first request to the source object storage system;

[0124] If the target data does not exist, forwarding the first request to the target object storage system;

[0125] The forwarding of the first request according to the second configuration information further includes:

[0126] In a case where the type of the first request is to read data, determining whether the target object storage system has target data corresponding to the first request according to an instruction of the second configuration;

[0127] If the target data exists, forwarding the first request to the target object storage system;

[0128] If the target data does not exist, the first request is forwarded to the source object storage system.

[0129] In this embodiment, in the process of forwarding the first request according to the first configuration information, data migration is performed synchronously, and the source object storage system is still the main service provider. Therefore, when the type of the first request is to read data, the target data is queried from the source object storage system first. If the target data exists, the first request is forwarded to the source object storage system; if not, the first request is forwarded to the target object storage system.

[0130] In contrast, in the process of forwarding the first request according to the second configuration information, the target object storage system is the main service provider, and at this time the target object storage system has begun to receive the data write service request initiated by the client, and some data may only exist in the target object storage system. Therefore, the target data is queried from the target object storage system first, and if the target data exists, the first request is forwarded to the target object storage system; if not, the first request is forwarded to the source object storage system.

[0131] In some embodiments of the present invention, forwarding the first request according to the first configuration information further includes:

[0132] In a case where the type of the first request is to upload data in segments, confirming whether the source object storage system has segment data corresponding to the first request according to an instruction of the first configuration;

[0133] If the source object storage system has the shard data corresponding to the first request, forwarding the first request to the source object storage system;

[0134] If the source object storage system does not have the shard data corresponding to the first request, forwarding the first request to the target object storage system;

[0135] The forwarding of the first request according to the second configuration information further includes:

[0136] In the case where the type of the first request is to upload data in pieces, confirming whether the target object storage system has piece data corresponding to the first request according to an instruction of the second configuration;

[0137] If the target object storage system has the shard data corresponding to the first request, forwarding the first request to the target object storage system;

[0138] If the target object storage system does not have the shard data corresponding to the first request, the first request is forwarded to the source object storage system.

[0139] In this embodiment, in the process of forwarding the first request according to the first configuration information, the source object storage system still serves as the main service provider, and it is prioritized to query from the source object storage system whether there is shard data corresponding to the first request; conversely, in the process of forwarding the first request according to the second configuration information, the target object storage system serves as the main service provider, and it is prioritized to query from the target object storage system whether there is shard data corresponding to the first request.

[0140] Then, the integrity of the data uploaded in slices needs to be ensured. Therefore, in the first configuration information, if the source object storage system has slice data corresponding to the first request, for example, other slice data that has been successfully uploaded already exists, then the service request needs to be forwarded to the source object storage system so that the slice data can be merged into complete data; if it does not exist, then it is the slice data uploaded for the first time, and then it can be forwarded to the target object storage system. The second configuration information is similar to the first configuration information, and will not be repeated here.

[0141] In actual applications, whether the type of the first request is to upload data in pieces can be determined by determining whether the service request includes uploadId, where uploadId is the serial number corresponding to the piece data when uploading data in pieces.

[0142] In some embodiments of the present invention, forwarding the first request according to the first configuration information further includes:

[0143] When the type of the first request is to check whether a container exists, or to obtain a data list corresponding to a container, forwarding the first request to the source object storage system according to an instruction of the first configuration;

[0144] The forwarding of the first request according to the second configuration information further includes:

[0145] In the case that the type of the first request is to check whether a container exists, or to obtain a data list corresponding to a container, the first request is forwarded to the target object storage system according to an instruction of the second configuration.

[0146] In this embodiment, in the process of forwarding the first request according to the first configuration information, the source object storage server acts as the main service provider, so the first request of the type of checking whether the container exists, or obtaining the data list corresponding to the container is forwarded to the source object storage server.

[0147] Relatively speaking, in the process of forwarding the first request according to the second configuration information, the target object storage server acts as the main service provider, so the first request of the type of checking whether the container exists, or obtaining the data list corresponding to the container is forwarded to the target object storage server.

[0148] In a specific implementation, if the first request includes a HeadBucket instruction, it can be confirmed that the type of the first request is to check whether the container exists; if the first request includes a ListObjects instruction, it can be confirmed that the type of the first request is to obtain a data list corresponding to the container.

[0149] In some embodiments of the present invention, forwarding the first request according to the first configuration information further includes:

[0150] When the type of the first request is to operate on a container, forwarding the first request to the target object storage system according to an instruction of the first configuration;

[0151] In a case where the type of the first request is to delete data, forwarding the first request to the source object storage system and the target object storage system according to an instruction of the first configuration;

[0152] The forwarding of the second request according to the second configuration information further includes:

[0153] When the type of the second request is to operate on a container, forwarding the second request to the target object storage system according to an instruction of the second configuration;

[0154] In the case that the type of the second request is to delete data, the second request is forwarded to the source object storage system and the target object storage system according to the instruction of the second configuration.

[0155] In this embodiment, since the first request type is to operate on the container, it will not affect the data consistency after data migration. Therefore, the configuration items of the forwarding rules for the first request type to operate on the container can be the same as those of the first configuration information and the second configuration information, both of which are to forward the first request to the target object storage system.

[0156] For the first request of type deleting data, bidirectional forwarding is performed on the source object storage system and the target object storage system, and both systems delete data at the same time, which can ensure data consistency. Therefore, the configuration items of the forwarding rule of type deleting data in the first configuration information and the second configuration information can also be the same, both of which forward the second request to the source object storage system and the target object storage system.

[0157] In some examples, the first configuration information set by the migration proxy gateway, that is, the configuration item of the proxy rule write_to_src, may be specifically as follows:

[0158] 1) All bucket operations are forwarded to the destination gateway.

[0159] 2) HeadBucket / ListObjects is forwarded to the source gateway.

[0160] 3) Create a new object (i.e. write data) and forward it to the source gateway.

[0161] 4) The read object is forwarded to the source gateway first, and if it does not exist, it is forwarded to the destination gateway.

[0162] 5) Requests with uploadId are first checked from the source gateway to see if they exist. If they exist, they are forwarded to the source gateway; otherwise, they are forwarded to the destination gateway.

[0163] 6) Delete operations are forwarded in both directions.

[0164] The second configuration information of the migration proxy gateway setting, that is, the configuration items of the proxy rule write_to_dest, can be specifically as follows:

[0165] 1) All bucket operations are forwarded to the destination gateway.

[0166] 2) HeadBucket / ListObjects is forwarded to the destination gateway.

[0167] 3) The newly created object is forwarded to the destination gateway.

[0168] 4) The read object is forwarded to the destination gateway first. If it does not exist, it is forwarded to the source gateway.

[0169] 5) Requests with uploadId are first checked from the destination gateway to see if they exist. If they exist, they are forwarded to the destination gateway; otherwise, they are forwarded to the source gateway.

[0170] 6) Delete operations are forwarded in both directions.

[0171] Step 107, comparing the difference between the data stored in the source object storage system and the data stored in the target object storage system, determining the difference data, and synchronizing the data between the source object storage system and the target object storage system according to the difference data;

[0172] In step 107, after the first configuration information is adjusted to the second configuration information and the first request is forwarded according to the second configuration information, no data has been written to the original object storage system. Then, the difference data in the target object storage system of the source object storage system can be synchronized, that is, incremental migration can be performed.

[0173] In some embodiments of the present invention, comparing the difference between the data stored in the source object storage system and the data stored in the target object storage system to determine the difference data includes:

[0174] Traversing each first data in the source object storage system and each second data stored in the target object storage system respectively;

[0175] If the target object storage system does not have the second data corresponding to the first data, determining the first data as the difference data;

[0176] If the target object storage system has the second data corresponding to the first data, verifying whether the first data and the second data are consistent;

[0177] If the first data and the second data are inconsistent, determining the first data as the difference data;

[0178] If the second data is data deleted in the source object storage system, the second data is determined as the difference data.

[0179] In this embodiment, if each first data in the source object storage system has been migrated, there will be corresponding and consistent second data in the target object storage system. However, since the source object storage system will still receive business requests for writing data during the process of forwarding the first request according to the first configuration information, the data in the source object storage system may be added, modified or deleted. Therefore, it is necessary to traverse the data in the source object storage system and the target object storage system respectively to confirm the difference data.

[0180] If the target object storage system does not have second data corresponding to the first data, the first data is determined as newly added data and as difference data;

[0181] If the target object storage system has second data corresponding to the first data, further verify whether the first data and the second data are consistent;

[0182] If the first data and the second data are inconsistent, the first data is determined to be modified data and is determined as difference data;

[0183] If the second data is data deleted in the source object storage system, the second data is determined as data that has not been deleted in time and is determined as differential data.

[0184] In some examples, incremental migration can be achieved through the scanning tool and synchronization tool in the above migration agent system. The details are as follows:

[0185] The scanning tool performs a full scan from the source object storage system while performing a full scan from the migration proxy system, and compares the scan results. If the object exists in the migration proxy system but not in the source object storage system, the object needs to be deleted; if it exists in the source object storage system but not in the migration proxy system, it means that the object is a newly added object; if the object's modification time, size, and data verification are inconsistent, it means that the object has been modified. All the above difference data must be recorded in the migration proxy system.

[0186] Then, use the synchronization tool to migrate. It should be noted that if there are still business requests for writing data being forwarded to the source gateway during this process, it is necessary to repeat the scanning synchronization until no business requests for writing data are forwarded to the source gateway.

[0187] In actual applications, after the incremental migration is completed, the DNS configuration of the bucket domain name can be changed from the IP address of the migration proxy gateway to the IP address of the destination gateway, that is, the interception of the first request is stopped, and all services of the source object storage system are migrated to the target object storage system. At this time, the flow of business requests is as follows: Figure 7 As shown in the figure, some clients can directly access the target object storage system, but because the DNS configuration takes time to take effect, some client business requests are still sent to the proxy gateway.

[0188] In some embodiments of the present invention, after synchronizing the source object storage system and the target object storage system according to the difference data, the method further includes:

[0189] Determine whether there is the first request of type reading data;

[0190] If the first request of the type of reading data does not exist, obtaining third configuration information, and adjusting the second configuration information to the third configuration information;

[0191] According to the instruction of the third configuration information, all the first requests are forwarded to the target object storage system.

[0192] In this embodiment, if there is no first request of the type of reading data, it means that there is no need to read data from the source object storage system, and the target object storage system can already provide services independently and completely. Therefore, the second configuration information is adjusted to the third configuration information, and according to the instruction of the third configuration information, all first requests are forwarded to the target object storage system.

[0193] The function of the third configuration information is similar to that of the first configuration information and the second configuration information, and will not be described in detail here. The difference is that the third configuration information only indicates to forward all first requests to the target object storage system, that is, all services are taken over by the target object storage system.

[0194] In a specific implementation, the third configuration information may be a proxy rule of the proxy server, and the proxy rule may be named all_to_dest, that is, all service requests are forwarded to the target object storage system.

[0195] In actual applications, when the IP address of the migration proxy gateway is changed to the IP address of the destination gateway and it is confirmed that no service requests have passed through the migration proxy gateway, the function of the migration proxy gateway to forward requests can be stopped to achieve complete migration of data and services. Figure 8 As shown, there is only a service request initiated by the client to the destination gateway.

[0196] Step 108: Feedback the result of migrating the data.

[0197] In step 108, after the full migration and incremental migration of the data are completed, the result of the data migration is fed back to the party requesting the data migration.

[0198] It should be noted that, for the sake of simplicity, the method embodiments are described as a series of action combinations, but those skilled in the art should be aware that the embodiments of the present invention are not limited by the order of the actions described, because according to the embodiments of the present invention, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present invention.

[0199] The embodiments of the present invention have the following advantages: the embodiments of the present invention intercept and forward the first request during the process of migrating data from the source object storage system to the target object storage system, forward the first request of the type of writing data to the source object storage system according to the indication of the first configuration information, adjust the first configuration information to the second configuration information when it is confirmed that the second request does not exist, forward the first request of the type of writing data to the source object storage system according to the indication of the second configuration information, compare the data difference between the source server and the destination server and synchronize the data, thereby ensuring data consistency without interrupting the related business of the source server, achieving smooth migration, and making the user unaware of the data migration process.

[0200] Reference Fig. 9 , shows a schematic diagram of the structure of a data processing device based on an object storage system provided by an embodiment of the present invention, which may specifically include the following modules:

[0201] The data migration module 901 is used to migrate the data in the source object storage system to the target object storage system in response to a request to migrate the data;

[0202] A first request interception module 902 is used to intercept a first request during the process of migrating data from a source object storage system to a target object storage system; wherein the first request is a service request initiated by a client to the source object storage system;

[0203] The first forwarding module 903 is used to obtain first configuration information and forward the first request according to the first configuration information, including: when the type of the first request is to write data, forwarding the first request to the source object storage system according to the instruction of the first configuration information;

[0204] A query module 904 is used to query whether there is a second request through the source object storage system during the process of forwarding the first request according to the first configuration information; wherein the second request is a service request received by the source object storage system from the client without being intercepted;

[0205] A first configuration adjustment module 905, configured to obtain second configuration information if the second request does not exist, and adjust the first configuration information to the second configuration information;

[0206] The second forwarding module 906 is configured to forward the first request according to the second configuration information, including: when the type of the first request is to write data, forwarding the first request to the target object storage system according to the instruction of the second configuration information;

[0207] The data synchronization module 907 is used to compare the difference between the data stored in the source object storage system and the data stored in the target object storage system, determine the difference data, and synchronize the data between the source object storage system and the target object storage system according to the difference data;

[0208] The result feedback module 908 is used to feedback the result of migrating data.

[0209] In some embodiments of the present invention, the first forwarding module 903 is further configured to:

[0210] a target data determination submodule, configured to determine whether the source object storage system has target data corresponding to the first request according to an instruction of the first configuration when the type of the first request is to read data;

[0211] A source forwarding first submodule, configured to forward the first request to the source object storage system if the target data exists;

[0212] a first destination forwarding submodule, configured to forward the first request to the target object storage system if the target data does not exist;

[0213] The second forwarding module 906 is further configured to:

[0214] In a case where the type of the first request is to read data, determining whether the target object storage system has target data corresponding to the first request according to an instruction of the second configuration;

[0215] If the target data exists, forwarding the first request to the target object storage system;

[0216] If the target data does not exist, the first request is forwarded to the source object storage system.

[0217] In some embodiments of the present invention, the first forwarding module 903 is further configured to:

[0218] In a case where the type of the first request is to upload data in segments, confirming whether the source object storage system has segment data corresponding to the first request according to an instruction of the first configuration;

[0219] If the source object storage system has the shard data corresponding to the first request, forwarding the first request to the source object storage system;

[0220] If the source object storage system does not have the shard data corresponding to the first request, forwarding the first request to the target object storage system;

[0221] The second forwarding module 906 is further configured to:

[0222] In the case where the type of the first request is to upload data in pieces, confirming whether the target object storage system has piece data corresponding to the first request according to an instruction of the second configuration;

[0223] If the target object storage system has the shard data corresponding to the first request, forwarding the first request to the target object storage system;

[0224] If the target object storage system does not have the shard data corresponding to the first request, the first request is forwarded to the source object storage system.

[0225] In some embodiments of the present invention, the first forwarding module 903 is further configured to:

[0226] When the type of the first request is to check whether a container exists, or to obtain a data list corresponding to a container, forwarding the first request to the source object storage system according to an instruction of the first configuration;

[0227] The second forwarding module 906 is further configured to:

[0228] In the case that the type of the first request is to check whether a container exists, or to obtain a data list corresponding to a container, the first request is forwarded to the target object storage system according to an instruction of the second configuration.

[0229] In some embodiments of the present invention, the first forwarding module 903 is further configured to:

[0230] When the type of the first request is to operate on a container, forwarding the first request to the target object storage system according to an instruction of the first configuration;

[0231] In a case where the type of the first request is to delete data, forwarding the first request to the source object storage system and the target object storage system according to an instruction of the first configuration;

[0232] The second forwarding module 906 is further configured to:

[0233] When the type of the second request is to operate on a container, forwarding the second request to the target object storage system according to an instruction of the second configuration;

[0234] In the case that the type of the second request is to delete data, the second request is forwarded to the source object storage system and the target object storage system according to the instruction of the second configuration.

[0235] In some embodiments of the present invention, the apparatus further comprises:

[0236] A first determination module, used for determining whether the first request of the type of reading data exists;

[0237] A second configuration information adjustment module, configured to obtain third configuration information and adjust the second configuration information to the third configuration information if the first request of the type of reading data does not exist;

[0238] According to the instruction of the third configuration information, all the first requests are forwarded to the target object storage system.

[0239] In some embodiments of the present invention, the data synchronization module 907 includes:

[0240] A traversal submodule, used to traverse each first data in the source object storage system and each second data stored in the target object storage system respectively;

[0241] A first difference data determination submodule, configured to determine the first data as the difference data if the second data corresponding to the first data does not exist in the target object storage system;

[0242] a verification submodule, configured to verify whether the first data and the second data are consistent if the target object storage system has the second data corresponding to the first data;

[0243] a second difference data determination submodule, configured to determine the first data as the difference data if the first data and the second data are inconsistent;

[0244] The third difference data determining submodule is configured to determine the second data as the difference data if the second data is data deleted in the source object storage system.

[0245] Some embodiments of the present invention further provide an electronic device, which may include a processor, a memory, and a computer program stored in the memory and capable of running on the processor, and when the computer program is executed by the processor, the method for data processing based on the object storage system as described above is implemented.

[0246] Some embodiments of the present invention further provide a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the method for data processing based on an object storage system as described above is implemented.

[0247] Some embodiments of the present invention further provide a computer program product, including a computer program, which implements the above method for data processing based on an object storage system when executed by a processor.

[0248] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0249] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0250] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0251] Those skilled in the art will appreciate that the embodiments of the present invention may be provided as methods, devices, or computer program products. Therefore, the embodiments of the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the embodiments of the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0252] The embodiments of the present invention are described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (systems), and computer program products according to the embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0253] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0254] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable terminal device to implement the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0255] Although the preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present invention.

[0256] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "including one..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the above elements.

[0257] The method, device, equipment and medium for data processing based on the object storage system are introduced in detail above. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as limiting the present invention.

Claims

1. A method for data processing based on an object storage system, characterized in that: The method comprises: In response to a request to migrate data, migrate the data in the source object storage system to the target object storage system; In the process of migrating data from a source object storage system to a target object storage system, intercepting a first request; wherein the first request is a service request initiated by a client to the source object storage system; Acquiring first configuration information, and forwarding the first request according to the first configuration information, includes: when the type of the first request is to write data, forwarding the first request to the source object storage system according to an instruction of the first configuration information; In the process of forwarding the first request according to the first configuration information, querying through the source object storage system whether there is a second request; wherein the second request is a service request received by the source object storage system from the client without being intercepted; If the second request does not exist, obtaining second configuration information, and adjusting the first configuration information to the second configuration information; Forwarding the first request according to the second configuration information includes: when the type of the first request is to write data, forwarding the first request to the target object storage system according to an instruction of the second configuration information; Comparing the difference between the data stored in the source object storage system and the data stored in the target object storage system, determining the difference data, and performing data synchronization on the source object storage system and the target object storage system according to the difference data; Provides feedback on the results of the migration data.

2. The method according to claim 1, characterized in that: The forwarding the first request according to the first configuration information further includes: In a case where the type of the first request is to read data, determining whether the source object storage system has target data corresponding to the first request according to an instruction of the first configuration; If the target data exists, forwarding the first request to the source object storage system; If the target data does not exist, forwarding the first request to the target object storage system; The forwarding of the first request according to the second configuration information further includes: In a case where the type of the first request is to read data, determining whether the target object storage system has target data corresponding to the first request according to an instruction of the second configuration; If the target data exists, forwarding the first request to the target object storage system; If the target data does not exist, the first request is forwarded to the source object storage system.

3. The method according to claim 1, characterized in that The forwarding the first request according to the first configuration information further includes: In the case where the type of the first request is to upload data in pieces, confirming whether the source object storage system has piece data corresponding to the first request according to the instruction of the first configuration; If the source object storage system has shard data corresponding to the first request, forwarding the first request to the source object storage system; If the source object storage system does not have the shard data corresponding to the first request, forwarding the first request to the target object storage system; The forwarding of the first request according to the second configuration information further includes: In the case where the type of the first request is to upload data in pieces, confirming whether the target object storage system has piece data corresponding to the first request according to an instruction of the second configuration; If the target object storage system has the shard data corresponding to the first request, forwarding the first request to the target object storage system; If the target object storage system does not have the shard data corresponding to the first request, the first request is forwarded to the source object storage system.

4. The method according to claim 1, characterized in that: The forwarding the first request according to the first configuration information further includes: When the type of the first request is to check whether a container exists, or to obtain a data list corresponding to a container, forwarding the first request to the source object storage system according to an instruction of the first configuration; The forwarding of the first request according to the second configuration information further includes: In the case that the type of the first request is to check whether a container exists, or to obtain a data list corresponding to a container, the first request is forwarded to the target object storage system according to an instruction of the second configuration.

5. The method according to any one of claims 1 to 4, characterized in that: The forwarding the first request according to the first configuration information further includes: When the type of the first request is to operate on a container, forwarding the first request to the target object storage system according to an instruction of the first configuration; In a case where the type of the first request is to delete data, forwarding the first request to the source object storage system and the target object storage system according to an instruction of the first configuration; The forwarding of the second request according to the second configuration information further includes: When the type of the second request is to operate on a container, forwarding the second request to the target object storage system according to an instruction of the second configuration; In the case that the type of the second request is to delete data, the second request is forwarded to the source object storage system and the target object storage system according to the instruction of the second configuration.

6. The method according to any one of claims 1 to 4, characterized in that: After synchronizing the source object storage system and the target object storage system according to the difference data, the method further includes: Determine whether there is the first request of type reading data; If the first request of the type of reading data does not exist, obtaining third configuration information, and adjusting the second configuration information to the third configuration information; According to the instruction of the third configuration information, all the first requests are forwarded to the target object storage system.

7. The method according to claim 1, characterized in that The comparing the difference between the data stored in the source object storage system and the data stored in the target object storage system to determine the difference data includes: Traversing each first data in the source object storage system and each second data stored in the target object storage system respectively; If the target object storage system does not have the second data corresponding to the first data, determining the first data as the difference data; If the target object storage system has the second data corresponding to the first data, verifying whether the first data and the second data are consistent; If the first data and the second data are inconsistent, determining the first data as the difference data; If the second data is data deleted in the source object storage system, the second data is determined as the difference data.

8. A data processing device based on an object storage system, characterized in that: The device comprises: A data migration module, configured to migrate data from a source object storage system to a target object storage system in response to a request to migrate data; A first request interception module is used to intercept a first request during the process of migrating data from a source object storage system to a target object storage system; wherein the first request is a service request initiated by a client to the source object storage system; A first forwarding module, configured to obtain first configuration information, and forward the first request according to the first configuration information, including: when the type of the first request is to write data, forwarding the first request to the source object storage system according to an instruction of the first configuration information; A query module, configured to query whether a second request exists through the source object storage system during the process of forwarding the first request according to the first configuration information; wherein the second request is a service request received by the source object storage system from the client without being intercepted; A first configuration adjustment module, configured to obtain second configuration information and adjust the first configuration information to the second configuration information if the second request does not exist; A second forwarding module, configured to forward the first request according to the second configuration information, comprising: when the type of the first request is to write data, forwarding the first request to the target object storage system according to an instruction of the second configuration information; A data synchronization module, used to compare the difference between the data stored in the source object storage system and the data stored in the target object storage system, determine the difference data, and synchronize the data between the source object storage system and the target object storage system according to the difference data; The result feedback module is used to feedback the results of the migration data.

9. An electronic device, characterized in that: The method comprises a processor, a memory, and a computer program stored in the memory and capable of running on the processor, wherein when the computer program is executed by the processor, the method for data processing according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method for data processing according to any one of claims 1 to 7 is implemented.

Images