Storage method, migration method, download method, storage system, electronic device, and medium

Abstract

Embodiments of the present application provide a storage method, a migration method, a download method, a storage system, an electronic device and a medium, wherein the storage method stores a storage object by using a data sharing network, different storage nodes in the data sharing network are connected in a point-to-point manner, and the storage method specifically comprises: storing the storage object; sending the storage object to a connected second storage node, so that the second storage node stores and / or forwards the storage object; after successfully storing the storage object, sending object information of the storage object to a record node, so that the record node saves storage records of the first storage node; and the storage records comprise object information stored by the corresponding storage node. Embodiments of the present application can reduce the dependence of object storage service quality on the reliability of the machine room, and further improve the object storage service quality.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to the technical field of storage, in particular to a storage method, a migration method, a download method, a storage system, an electronic device and a medium. BACKGROUND

[0002] With the rapid development of Internet technology, Internet produces a large amount of data every minute, such as pictures, documents, audio and video, forms, etc. Most of these data are unstructured data, and object storage is a widely used storage technology for unstructured data storage.

[0003] In today's era of rapid data growth, a large number of enterprises, government agencies, educational institutions, etc. have their own one or more data centers, which undertake the task of centralized storage and centralized management of data to provide object storage services. At present, data centers are usually deployed in a cluster manner in a single or multiple machine rooms.

[0004] In actual application, the quality of object storage services provided by the data center is easily affected by the reliability of the machine room. For example, when the machine room has power failure, network failure and other faults, the data center may not be able to provide object storage services, or the quality of object storage services provided by the data center is low. SUMMARY

[0005] Embodiments of the present application aim to provide a storage method, a migration method, a download method, a storage system, an electronic device and a medium to reduce the dependence of the quality of object storage services on the reliability of the machine room, thereby improving the quality of object storage services.

[0006] The specific technical solutions are as follows:

[0007] In a first aspect of the present application, a storage method is provided, which stores storage objects using a data sharing network, different storage nodes in the data sharing network are connected point-to-point, and the method is applied to a first storage node and includes:

[0008] storing storage objects;

[0009] sending the storage objects to a connected second storage node to enable the second storage node to store and / or forward the storage objects;

[0010] after successfully storing the storage objects, sending object information of the storage objects to a record node to enable the record node to save storage records of the first storage node; the storage records include object information stored by the corresponding storage node.

[0011] In a second aspect of the present application, a migration method is provided, which is used for storing objects in a data sharing network, different storage nodes in the data sharing network are connected point-to-point, and the method comprises:

[0012] The recording node receives a data migration request.

[0013] The recording node determines a third storage node and a fourth storage node in the data sharing network according to the data migration request.

[0014] The recording node updates the storage record of the third storage node to the storage record of the fourth storage node; the storage record comprises object information stored by a corresponding storage node.

[0015] The fourth storage node obtains the storage record from the recording node; the storage record comprises object information stored by a corresponding storage node.

[0016] If the fourth storage node does not store a first storage object corresponding to the first object information in the storage record locally, the fourth storage node obtains the first storage object from a fifth storage node in the data sharing network.

[0017] In a third aspect of the present application, a download method is provided, which is applied to a sixth storage node, and the method comprises:

[0018] Receiving a download request; the download request comprises second object information.

[0019] If a second storage object corresponding to the second object information is not stored locally, the second storage object is obtained from a seventh storage node; the sixth storage node and the seventh storage node are connected point-to-point.

[0020] In a fourth aspect of the present application, a storage system is provided, which comprises a recording node and a data sharing network.

[0021] The data sharing network comprises a plurality of storage nodes, and different storage nodes are connected point-to-point.

[0022] The recording node comprises:

[0023] A storage record of the storage node is saved by the saving module; the storage record comprises object information stored by a corresponding storage node.

[0024] The migration request receiving module is used for receiving a data migration request.

[0025] The migration node determining module is used for determining a third storage node and a fourth storage node in the data sharing network according to the data migration request.

[0026] a migration module, configured to update the storage record of the third storage node to the storage record of the fourth storage node; the storage record comprises object information stored by a corresponding storage node;

[0027] The fourth storage node is located in the data sharing network, and is configured to obtain the storage record from the record node, and obtain the first storage object from a fifth storage node in the data sharing network if the first storage object corresponding to the first object information in the storage record is not stored locally.

[0028] In a fifth aspect of the embodiments of the present application, a computer readable storage medium is further provided, which stores instructions, and when the instructions are run on a computer, the computer is caused to execute the method according to any one of the preceding aspects.

[0029] In a sixth aspect of the embodiments of the present application, a computer program product is further provided, which contains instructions, and when the instructions are run on a computer, the computer is caused to execute the method according to any one of the preceding aspects.

[0030] The storage method, the migration method, the download method, the storage system, the electronic device and the medium provided by the embodiments of the present application store storage objects by using a data sharing network, different storage nodes in the data sharing network are connected point-to-point, and the first storage node can send a storage object to the connected second storage node, so that distributed storage of the storage object and reliability of object storage can be realized, thereby reducing the dependence of object storage service quality on the reliability of a machine room, and improving the object storage service quality.

[0031] In addition, the embodiments of the present application store object information successfully stored by the first storage node via the record node, so that in the case that the first storage node cannot normally provide services due to failure or other reasons, the storage object successfully stored by the first storage node can be migrated to other storage nodes, thereby further improving the reliability of data storage. BRIEF DESCRIPTION OF DRAWINGS

[0032] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or the prior art description will be briefly introduced.

[0033] Figure 1 FIG. 1 is a structural schematic diagram of a data sharing network according to an embodiment of the present application;

[0034] Figure 2 FIG. 3 is a step flowchart of a storage method according to an embodiment of the present application;

[0035] Figure 3 FIG. 4 is a step flowchart of a migration method according to an embodiment of the present application;

[0036] Figure 4 Step flow chart of migration method for an embodiment of the present application;

[0037] Figure 5 Step flow chart of migration method for an embodiment of the present application;

[0038] Figure 6 Step flow chart of download method for an embodiment of the present application;

[0039] Figure 7 Structural schematic diagram of storage system for an embodiment of the present application;

[0040] Figure 8 Structural schematic diagram of storage system for an embodiment of the present application;

[0041] Figure 9 Step flow chart of upload method for an embodiment of the present application;

[0042] Figure 10 Step flow chart of download method for an embodiment of the present application;

[0043] Figure 11 Step flow chart of migration method for an embodiment of the present application;

[0044] Figure 12 Structural schematic diagram of storage device for an embodiment of the present application;

[0045] Figure 13 Structural schematic diagram of migration device for an embodiment of the present application;

[0046] Figure 14 Structural schematic diagram of migration device for an embodiment of the present application;

[0047] Figure 15 Structural block diagram of electronic device for an embodiment of the present application. DETAILED DESCRIPTION

[0048] The technical solutions in the embodiments of the present application will be described below with reference to the drawings in the embodiments of the present application.

[0049] The embodiments of the present application provide an object storage scheme based on a data sharing network. A data sharing network across operators and regions can be constructed by using point-to-point connection technology. The connection in the embodiments of the present application can refer to data interaction through a wired network or a wireless network.

[0050] With reference to Figure 1Fig. 1 shows a structural schematic diagram of a data sharing network according to an embodiment of the present application. The data sharing network can include storage node a, storage node b, storage node c, storage node d, storage node e and storage node f. Any two storage nodes in the data sharing network can be connected in a point-to-point manner.

[0051] It can be understood that Figure 1 The data sharing network shown in the figure includes six storage nodes only as an example. In fact, a person skilled in the art can determine the number N of storage nodes in the data sharing network according to actual application requirements. For example, N can be a natural number greater than 6, or the order of magnitude of N can be thousands, tens of thousands, or even hundreds of thousands, millions, etc.

[0052] Figure 1 The connection relationship between the storage nodes in the data sharing network shown in the figure is also an example. In fact, a storage node can be connected with other storage nodes according to actual conditions. For example, the first storage node can discover the second storage node by using a node discovery method and be connected with the second storage node.

[0053] In actual applications, N devices connected to the Internet can be prepared, one of which can be used as a storage node. The storage devices such as disks and / or memories in a storage node can be used to store storage objects. The fusion of the storage devices of all storage nodes can determine the storage capacity that the object storage service can provide. Generally speaking, the larger the number N of storage nodes, the stronger the storage capacity that can be provided. In addition, the storage node can also provide upload and download services based on the stored storage objects and the data sharing network in which it is located.

[0054] In actual applications, a preset application (such as a data sharing application) supporting a point-to-point connection technology such as an InterPlanetary File System (IPFS) or BitTorrent (BT) can be used to construct a data sharing network. In specific implementation, a preset application is run on a device, and the device can be used as a storage node in the data sharing network.

[0055] The embodiment of the present application further provides a storage method, which can store a storage object by using a data sharing network, different storage nodes in the data sharing network are connected in a point-to-point manner, the method can be applied to a first storage node, and specifically can include the following steps:

[0056] In conclusion, in the embodiment of the present application, different storage nodes are connected in a point-to-point manner, in actual application, there are many such connection node pairs (for example, the connection node pair corresponding to the first storage node and the second storage node). Moreover, the first storage node sends a storage object to a connected second storage node, so that the second storage node stores and / or forwards the storage object; in this way, multiple storage nodes can store the same storage object, and thus the distributed storage of the storage object and the reliability of the object storage can be realized. In the case of realizing the distributed storage of the storage object, the embodiment of the present application can reduce the dependence of the object storage service quality on the reliability of the machine room, and thus the object storage service quality can be improved.

[0057] Moreover, after successfully storing the storage object, the embodiment of the present application sends object information of the storage object to a record node, so that the record node saves the storage record of the first storage node. Since the embodiment of the present application stores the object information successfully stored by the first storage node via the record node, in the case that the first storage node cannot normally provide services due to faults or the like, the storage object successfully stored by the first storage node is migrated to other storage nodes, and thus the reliability of data storage can be further improved.

[0058] The embodiment of the present application is described below by means of specific examples.

[0059] Reference is made to Figure 2 FIG. 1 shows a step flowchart of a storage method according to an embodiment of the present application, the method can store a storage object by using a data sharing network, different storage nodes in the data sharing network are connected in a point-to-point manner, the method can be applied to a first storage node, and specifically can include the following steps:

[0060] Step 201, storing a storage object;

[0061] Step 202, sending the storage object to a connected second storage node, so that the second storage node stores and / or forwards the storage object;

[0062] Step 203, after successfully storing the storage object, sending object information of the storage object to the record node, so that the record node saves the storage record of the first storage node; the storage record can include object information stored by the corresponding storage node.

[0063] Figure 2 The method embodiment is used for describing the processing procedure of the first storage node.

[0064] In step 201, the storage object can be a picture, a document, an audio / video, a form, etc. In actual application, the storage object can correspond to a file, so storing the storage object usually means storing the file corresponding to the storage object.

[0065] In actual application, the storage object can come from a user. For example, the storage object can be a storage object corresponding to a user's uploading request. Of course, the storage object can also come from other storage nodes connected to the first storage node.

[0066] The data sharing network of the embodiment of the application can include a large number of storage nodes. In theory, any storage node can have the providing capability of object storage service, uploading service and downloading service.

[0067] In one implementation manner of the application, in the case of receiving an uploading request, a storage node can be randomly specified as a target storage node corresponding to the uploading request, and the target storage node processes the uploading request.

[0068] In another implementation manner of the application, in the case of receiving an uploading request, the geographical position information corresponding to the uploading request and the geographical position information corresponding to the storage node can be matched, and thus the storage node matched successfully can be used as the target storage node. The geographical position information corresponding to the uploading request and the geographical position information corresponding to the target storage node are matched, which means that the distance between them is short. Generally, the distance is proportional to the network delay, so the embodiment of the application processes the uploading request by the target storage node matched with the geographical position information corresponding to the uploading request, which can improve the processing efficiency of the uploading request.

[0069] Therefore, in one embodiment of the application, the first storage node can be used as the target storage node to process the uploading request. The geographical position information corresponding to the uploading request is matched with the geographical position information corresponding to the first storage node. In this case, the storage object corresponding to the uploading request can be stored.

[0070] In actual application, storing the storage object can include caching the storage object or storing the storage object to a disk.

[0071] In a specific implementation, the storage manner corresponding to the storage object can include: sequential storage, chained storage, indexed storage, or hash storage, etc.

[0072] The hash storage process can specifically include: determining a hash value corresponding to the storage object; saving a mapping relationship between the object identifier of the storage object and the hash value; and storing the storage object according to the hash value. The hash value can be used to determine the storage address of the storage object, and thus the addressing of the storage object can be implemented.

[0073] In an example, after receiving the storage object uploaded by the user, a hash value can be calculated for the object content of the storage object, and a mapping relationship between the hash value and the object identifier can be stored in the database. The object identifier can be information such as an object name. Since different object contents can correspond to different hash values, the storage object is stored according to the hash value in the embodiment of the application, which can avoid repeated storage of the same object content on one storage node, and thus the storage space can be saved. In addition, the hash storage has the advantage of fast access speed.

[0074] In step 202, the storage nodes connected to the first storage node can be one or more. In actual application, the storage object can be sent to all or part of the connected storage nodes. For example, the second storage node can be selected from the storage nodes connected to the first storage node according to a backup threshold M (M can be a positive integer), and the number of the second storage node can be less than or equal to the backup threshold M.

[0075] The second storage node can store the storage object, and / or can forward the storage object to other storage nodes to store the storage object, and thus the distributed storage of the storage object can be implemented.

[0076] In actual application, the storage object and the object identifier can be sent to the second storage node to enable the second storage node to perform corresponding processing.

[0077] In step 203, after the storage object is successfully stored, the object information of the storage object is sent to the record node, and the object information can include the object identifier or the hash value.

[0078] The record node can be used to save the storage record of the storage node such as the first storage node, and the storage record can include the object information stored by the corresponding storage node.

[0079] In summary, the storage method of the embodiment of the present application, the first storage node sends the storage object to the connected second storage node, so that the second storage node stores and / or forwards the storage object; in this way, the same storage object can be stored by multiple storage nodes, so that the distributed storage of the storage object and the reliability of the object storage can be realized. In the case of realizing the distributed storage of the storage object, the embodiment of the present application can reduce the dependence of the object storage service quality on the reliability of the machine room, and thus improve the object storage service quality.

[0080] In addition, since the embodiment of the present application stores the object information successfully stored by the first storage node via the record node, in the case that the first storage node cannot normally provide services due to failure or the like, the storage object successfully stored by the first storage node is migrated to other storage nodes, and thus the reliability of data storage can be further improved.

[0081] The processing process of the second storage node is described below.

[0082] In actual application, the second storage node can receive the storage object sent by the first storage node, and store and / or forward the storage object.

[0083] For example, the second storage node can determine whether to store the storage object according to the remaining capacity of the storage space. For example, if the remaining capacity is less than a preset capacity, the storage object is stored, otherwise, the storage object is forwarded. Of course, in the case of storing the storage object, the storage object can also be forwarded.

[0084] The second storage node can store the storage object, and after successfully storing the storage object, send the object information of the storage object to the record node, so that the record node saves the storage record of the second storage node. For details, refer to steps 201 and 203, which are not described here.

[0085] Whether the second storage node stores the storage object or not, the second storage node can send the storage object to other connected storage nodes, so that the other storage nodes store and / or forward the storage object.

[0086] The processing process of the record node is described below.

[0087] Referring to Figure 3 , a step flow chart of a migration method of an embodiment of the present application is shown, the method can store the storage object by using a data sharing network, different storage nodes in the data sharing network are connected point-to-point, the method can be applied to the record node, and specifically can include the following steps:

[0088] Step 301, receiving a data migration request;

[0089] Step 302, determining a third storage node and a fourth storage node according to the data migration request;

[0090] Step 303, updating the storage record of the third storage node into the storage record of the fourth storage node; the storage record can include object information stored by the corresponding storage node.

[0091] The embodiment can be used to update the storage record of the third storage node into the storage record of the fourth storage node, so that the fourth storage node migrates the storage object of the third storage node according to the updated storage record, thereby solving the problem of data loss caused by the third storage node failure or offline (exiting the data sharing network).

[0092] In step 301, the data migration request can include the identification of the third storage node to request the migration of the storage record of the third storage node.

[0093] The third storage node can represent a failed storage node or an offline storage node. The polling method or the heartbeat detection method can be used to determine whether a storage node is a third storage node.

[0094] The polling method can periodically send inquiry information to the storage node, and the storage node can return state information. If no state information is received within a predetermined time period after sending the inquiry information, or the returned state information does not meet the requirements, it can be considered that the storage node has failed or is offline, so the storage node can be considered as a third storage node.

[0095] The heartbeat detection method can periodically send registration information by the storage node. If no registration information is received within a predetermined time period after receiving the registration information once, or the received registration information does not meet the requirements, it can be considered that the storage node has failed or is offline, so the storage node can be considered as a third storage node.

[0096] In an implementation manner, the third storage node can be determined by the management node. Specifically, the storage node can periodically send registration information to the management node, for example, the sending period of the registration information can be 5 minutes or the like. The management node saves the registration information of the storage node, and reads the registration information of the storage node at regular intervals (such as every 1 hour). If an offline third storage node (for example, no registration information is sent for more than 2 hours) is found, the management node sends a data migration request to the recording node, and the data migration request can include the identification of the third storage node and the like.

[0097] In step 302, the third storage node can be determined according to the identification of the third storage node contained in the data migration request.

[0098] The state of the storage node can include an offline state or an online state. One storage node in the online state can be randomly designated as the fourth storage node.

[0099] Alternatively, the geographic location information corresponding to the third storage node can be matched with the geographic location information corresponding to the storage node, and the storage node that matches successfully can be taken as the fourth storage node.

[0100] The geographic location information corresponding to the third storage node matches the geographic location information corresponding to the fourth storage node, specifically, can include that the difference between the geographic location information corresponding to the third storage node and the geographic location information corresponding to the fourth storage node meets a preset condition, or the third storage node and the fourth storage node correspond to the same geographic region.

[0101] The skilled in the art can determine the granularity of the geographic region according to actual needs. For example, for a city, the granularity of the geographic region can specifically include province, city, district, street, community, etc.; for a rural area, the granularity of the geographic region can specifically include province, city, county, township, village, etc.

[0102] In actual application, the fourth storage node can be determined in the order from fine to coarse granularity of the geographic region. For example, the fourth storage node can be first obtained from the first geographic region corresponding to the third storage node, the first geographic region has a finer granularity, for example, the granularity of the first geographic region is community or village. If the fourth storage node fails to be obtained, the granularity can be coarsened based on the first geographic region, for example, the granularity of the first geographic region is adjusted to street or township, and then the fourth storage node is obtained from the first geographic region corresponding to the third storage node. In this way, the fourth storage node is obtained until it is successfully obtained.

[0103] In step 303, the storage record can include object information stored by the corresponding storage node. For example, the correspondence between the identification of the storage node and the hash value of the stored object can be saved. Specifically, the correspondence can be saved by using a database file or an office file. For another example, a data table corresponding to a storage node can be saved, and the data table includes the stored object information.

[0104] The updating of the storage record of the third storage node into the storage record of the fourth storage node can specifically include: merging the first storage record of the third storage node into the second storage record of the fourth storage node. It can be understood that, in the merging process, if the first storage record is the same as the second storage record, the first storage record can be discarded.

[0105] In conclusion, the migration method of the embodiment of the application can update the storage record of the third storage node into the storage record of the fourth storage node, so that the fourth storage node can migrate the storage object of the third storage node according to the updated storage record, thereby solving the problem of data loss caused by the failure or offline (exiting the data sharing network) of the third storage node.

[0106] The processing process of the fourth storage node is described below.

[0107] Referring to Figure 4 , a step flowchart of a migration method of an embodiment of the application is shown, the method can store storage objects by using a data sharing network, different storage nodes in the data sharing network are connected point-to-point, the method can be applied to a fourth storage node, and specifically can include the following steps:

[0108] Step 401: obtaining a storage record from a record node; the storage record can include object information corresponding to a storage node;

[0109] Step 402: if a first storage object corresponding to the first object information in the storage record is not stored locally, obtaining the first storage object from a fifth storage node; the fourth storage node and the fifth storage node are connected point-to-point.

[0110] In this embodiment, the fourth storage node obtains the first storage object from the connected fifth storage node in the case that the first storage object corresponding to the first object information in the storage record is not stored locally, so that the storage object stored locally matches the storage record, and thus the reliability of object storage can be improved.

[0111] In addition, in the case that the third storage node fails or is offline (exits the data sharing network), the storage record of the third storage node can be updated into the storage record of the fourth storage node, so that the fourth storage node migrates the storage object of the third storage node according to the updated storage record, thereby solving the problem of data loss caused by the offline (exiting the data sharing network) of the third storage node.

[0112] In step 401, the fourth storage node can periodically obtain the storage record from the record node. For example, the fourth storage node can obtain the storage record from the record node every 12 hours.

[0113] The storage record stored by the record node can be updated according to the successful storage of the storage object by the fourth storage node, and / or the storage record stored by the record node can be updated according to the data migration request. Therefore, the fourth storage node can periodically acquire the updated storage record from the record node.

[0114] In step 402, if the first storage object corresponding to the first object information in the storage record is not stored locally, the first storage object can be acquired from the fifth storage node, so that the storage object stored locally matches the storage record, thereby improving the reliability of object storage.

[0115] In actual application, the hash value of the storage object stored successfully locally can be matched with the hash value in the storage record. If the matching is successful, it can be considered that the storage object corresponding to the hash value in the storage record is stored locally. Or, if the matching fails, it can be considered that the first storage object corresponding to the hash value in the storage record is not stored locally.

[0116] In actual application, the storage nodes connected to the fourth storage node can be one or more. In actual application, the inquiry can be sent to all or part of the connected storage nodes. For example, the fifth storage node can be selected from the storage nodes connected to the fourth storage node. If the first storage object is stored locally in the fifth storage node, the first storage object can be returned to the fourth storage node. Or, if the first storage object is not stored locally in the fifth storage node, the corresponding notification message can be returned to the fourth storage node, or the fifth storage node can acquire the first storage object from the eighth storage node connected.

[0117] In an example of the present application, the fourth storage node can send an inquiry to at least one fifth storage node connected, and the inquiry is used to inquire whether the fifth storage node stores the first storage object. If yes, the first storage object can be acquired from the fifth storage node. If the fifth storage node does not store the first storage object, the fifth storage node can send an inquiry to at least one eighth storage node connected, until the ninth storage node storing the first storage object is found. In actual application, the identification information of the ninth storage node and the like can be sent to the fourth storage node, so that the fourth storage node is connected to the ninth storage node and acquires the first storage object from the ninth storage node.

[0118] In specific implementation, if the first storage object corresponding to the first object information in the storage record is not stored locally, the first storage object is acquired from the fifth storage node; the fourth storage node and the fifth storage node are connected point-to-point.

[0119] In actual application, the object information in the storage record can be traversed, and after it is determined that the storage objects corresponding to the object information in the storage record are all stored by the fourth storage node, it can be considered that the migration ends.

[0120] To sum up, the migration method of the embodiment of the application can obtain the first storage node from the connected fifth storage node in the case that the fourth storage node does not store the first storage object corresponding to the first object information in the storage record locally, so as to make the storage objects stored locally match the storage record, and thus the reliability of object storage can be improved.

[0121] In addition, in the case that the third storage node fails or is offline (exits the data sharing network), the storage record of the third storage node can be updated to the storage record of the fourth storage node, so that the fourth storage node migrates the storage objects of the third storage node according to the updated storage record, and thus the problem of data loss caused by the third storage node being offline (exiting the data sharing network) can be solved.

[0122] The complete migration process is described below.

[0123] Referring to Figure 5 , a step flowchart of a migration method of an embodiment of the application is shown, the method can store storage objects by using a data sharing network, different storage nodes in the data sharing network are connected point-to-point, and the method can specifically include the following steps:

[0124] Step 501, the record node receives a data migration request;

[0125] Step 502, the record node determines a third storage node and a fourth storage node in the data sharing network according to the data migration request;

[0126] Step 503, the record node updates the storage record of the third storage node to the storage record of the fourth storage node; the storage record can include object information stored by the corresponding storage node;

[0127] Step 504, the fourth storage node obtains the storage record from the record node; the storage record includes object information stored by the corresponding storage node;

[0128] Step 505, the fourth storage node obtains the first storage object from a fifth storage node in the data sharing network in the case that the fourth storage node does not store the first storage object corresponding to the first object information in the storage record locally.

[0129] The processing process of the sixth storage node is described below.

[0130] Referring to Figure 6, a step flow chart of a downloading method of an embodiment of the present application is shown, the method is used for storing objects in a data sharing network, different storage nodes in the data sharing network are connected point to point, the method can be applied to a sixth storage node, and specifically can include the following steps:

[0131] Step 601, receiving a downloading request; the downloading request can include second object information;

[0132] Step 602, if a second storage object corresponding to the second object information is not stored locally, obtaining the second storage object from a seventh storage node; the sixth storage node and the seventh storage node are connected point to point.

[0133] In the embodiment of the present application, the sixth storage node can provide a downloading service. In step 601, a downloading request sent by a user can be received, and the downloading request can include second object information, such as an object identifier of a second object.

[0134] In an implementation manner, after receiving the downloading request, geographical position information corresponding to the downloading request and geographical position information corresponding to the storage node can be matched, so that a storage node that is successfully matched can be used as the sixth storage node. The geographical position information corresponding to the downloading request is matched with the geographical position information corresponding to the sixth storage node, which indicates that the distance between the two is relatively short. Generally, the distance is in a positive correlation with network delay, so the sixth storage node that is matched with the geographical position information corresponding to the downloading request processes the downloading request, which can improve the processing efficiency of the downloading request.

[0135] In another implementation manner, the sixth storage node can be obtained according to the second object information and storage records of the sixth storage node, so that the sixth storage node stores the second storage object corresponding to the second object information, and the processing efficiency of the downloading request can be improved. Specifically, the second object information can be matched with the storage records, if the second object information is matched with the storage records of a certain storage node, the storage node can be used as the sixth storage node.

[0136] Of course, the sixth storage node can be determined according to the second object information and geographical position information corresponding to the downloading request. Specifically, the sixth storage node can be determined according to first matching information between the geographical position information corresponding to the downloading request and the geographical position information corresponding to the storage node, and second matching information between the second object information and the storage records of the storage node. In this case, the geographical position information corresponding to the sixth storage node is matched with the geographical position information corresponding to the downloading request, and the storage records of the sixth storage node are also matched with the second object information.

[0137] In step 602, the sixth storage node can determine whether the second storage object corresponding to the second object information is stored locally, and if so, obtain the second storage object from the local storage and return the second storage object to the user.

[0138] If the hash storage mode is used, the corresponding determination process can include: according to the object identifier of the second storage object in the download request, searching in the mapping relationship between the object identifier of the storage object and the hash value to obtain the target hash value of the second storage object, and determining whether the second storage object corresponding to the target hash value is stored in the local storage.

[0139] If the second storage object corresponding to the second object information is not stored locally, the second storage object can be obtained from the seventh storage node by using the data sharing network. For the process of the sixth storage node obtaining the second storage object from the seventh storage node, since it is similar to the process of the fourth storage node obtaining the first storage object from the fifth storage node, it will not be repeated here, and mutual reference can be made.

[0140] In summary, the download method of the embodiment of the application can provide download service by obtaining the second storage object from the seventh storage node in the case that the second storage object corresponding to the second object information of the download request is not stored locally.

[0141] The embodiment of the application also provides a storage system, which refers to Figure 7 The storage system can specifically include: a recording node 701 and a data sharing network 702.

[0142] The data sharing network 702 can specifically include: a plurality of storage nodes 721, such as a first storage node, a second storage node, a third storage node, a fourth storage node, etc.; different storage nodes 721 are connected point-to-point. The recording node 701 saves the storage record of the above-mentioned storage node; the storage record can specifically include: object information stored by the corresponding storage node.

[0143] The recording node 701 can specifically include:

[0144] The saving module 711 is configured to save the storage record of the storage node; the storage record includes: object information stored by the corresponding storage node.

[0145] The migration request receiving module 712 is configured to receive a data migration request.

[0146] The migration node determining module 713 is configured to determine a third storage node and a fourth storage node in the data sharing network according to the data migration request.

[0147] The migration module 714 is configured to update the storage record of the third storage node to the storage record of the fourth storage node; the storage record includes object information corresponding to the storage of the storage node.

[0148] The fourth storage node 721 is located in the data sharing network, and is configured to obtain a storage record from the record node, and in a case where a first storage object corresponding to first object information in the storage record is not stored locally, obtain the first storage object from a fifth storage node in the data sharing network

[0149] In an implementation manner, the data sharing network can specifically include a first storage node.

[0150] The first storage node can specifically include:

[0151] The storage module is configured to store the storage object.

[0152] The first sending module is configured to send the storage object to a connected second storage node, so that the second storage node stores and / or forwards the storage object.

[0153] The second sending module is configured to, after successfully storing the storage object, send object information of the storage object to the record node, so that the record node saves a storage record of the first storage node; the storage record includes object information corresponding to the storage of the storage node.

[0154] In another implementation manner, the data sharing network can include a sixth storage node.

[0155] The sixth storage node can specifically include:

[0156] The download request receiving module is configured to receive a download request; the download request includes second object information.

[0157] The storage object obtaining module is configured to, in a case where a second storage object corresponding to the second object information is not stored locally, obtain the second storage object from a seventh storage node.

[0158] In another implementation manner, the storage system can further include a scheduling node configured to determine a corresponding storage node for an upload request or a download request.

[0159] In another implementation manner, the storage system can further include a management node configured to receive registration information of a storage node, and determine whether the storage node is offline according to the registration information.

[0160] In another implementation manner, the storage system can further include a database configured to save a mapping relationship between object identifiers and hash values of storage objects.

[0161] In actual application, the functions of any one of the scheduling node, the management node and the recording node can be realized via a separate device, or multiple functions of the scheduling node, the management node and the recording node can be integrated on one device.

[0162] In other implementation manners, a multi-tenant mode can also be supported, and data of different users is isolated by tenant to avoid mutual interference.

[0163] In actual application, the scheduling node, the management node and the recording node can be deployed centrally or distributedly, and the distributed deployment can reduce the dependence on the machine room database and further improve the reliability of the object storage service.

[0164] Referring to Figure 8 , a structure schematic diagram of a storage system of one embodiment of the present application is shown, which can specifically include a scheduling node, a recording node, a management node and a data sharing network. Figure 8 The storage node a, the storage node b and the storage node c of the data sharing network are shown in , and other storage nodes can be referred to each other, and in specific implementation, a node application can be deployed on the storage node, and the node application can include but is not limited to a gateway application, an encryption and decryption application, a content storage application and a data sharing application, etc.

[0165] The gateway application is used to provide an operation interface of the storage service, including user legality verification, etc.

[0166] The encryption and decryption application is used to provide at least one data encryption and decryption mode, such as symmetric / asymmetric encryption and decryption, full content encryption, partial content encryption, etc.

[0167] The content storage application is used to realize the functions of querying, uploading service and downloading service of the storage object, and sending registration information to the management node, etc.

[0168] The data sharing application is used to dynamically construct a peer-to-peer data sharing network, and the data sharing network can be a private network or a public network. The data sharing application can also share the storage network in the data sharing network. For example, in step 402 or step 502, the storage object is obtained from the connected storage node.

[0169] The scheduling node, the recording node and the management node can be located on the server. The scheduling node is used to provide the scheduling function of the storage node, such as the location-based nearest scheduling function or the scheduling function based on the storage record, etc. The content storage application of the storage node will register with the scheduling node regularly, and the scheduling service saves the registration information of the storage node to the database.

[0170] The management node detects the storage node in an offline state (such as a storage node that has not registered for the last two hours) by checking the registration information in the database in a timing manner, and sends a data migration request to the record node for the storage node in the offline state.

[0171] The record node is responsible for managing the storage record of the storage node, and the storage record can include object information stored by the storage node. After the storage node successfully saves the storage object, the record node can be notified to update the storage record. In the case of receiving the data migration request sent by the management node, a target storage node in the same geographic area can be selected for the source storage node of data migration, and then the storage record of the source storage node is merged into the storage record of the target storage node.

[0172] The following describes the processing process of the upload request.

[0173] Referring to Figure 9 , a step flowchart of an upload method of an embodiment of the application is shown, which can specifically include the following steps:

[0174] Step 901, the user equipment sends a node query request to the scheduling node;

[0175] Step 902, the scheduling node returns the node information to the user;

[0176] The user equipment can perform data interaction with a target storage node corresponding to the node information (such as the access address of the storage node), and the following gateway application, encryption and decryption application, content storage application and data sharing application can all be applications of the target storage node.

[0177] Step 903, the user equipment sends a file to be uploaded to the gateway application;

[0178] Step 904, the gateway application sends a file encryption request to the encryption and decryption application;

[0179] Step 905, the encryption and decryption application performs encryption processing on the file;

[0180] Step 906, the encryption and decryption application returns the ciphertext to the gateway application;

[0181] Step 907, the gateway application sends the ciphertext to the content storage application, so that the content storage application stores the ciphertext;

[0182] Step 908, the content storage application locally caches the ciphertext;

[0183] Step 909, the content storage application returns a second result to the gateway application;

[0184] Step 910, the gateway application returns a third result to the user equipment;

[0185] Step 911, the content storage application calculates a hash value for the ciphertext, and also stores a mapping relationship between the object identifier and the hash value in the database;

[0186] Step 912, the content storage application sends the ciphertext to the connected storage node by using the data sharing application;

[0187] The content storage application can send the ciphertext to other storage nodes for storage by calling the data sharing application according to a storage strategy.

[0188] Step 913, after the content storage application completes storage or the data sharing application sends the ciphertext to other storage nodes, a record update request is sent to the record node;

[0189] Step 914, the data sharing application returns a first result to the content storage application after completing the distribution of the ciphertext.

[0190] The following describes the processing process of the download request.

[0191] Referring to Figure 10 , a step flowchart of a download method according to an embodiment of the present application is shown, which can specifically include the following steps:

[0192] Step 1001, a user equipment sends a node query request to a scheduling node;

[0193] Step 1002, the scheduling node returns node information to the user;

[0194] Step 1003, the user equipment sends a download request to a gateway application;

[0195] Step 1004, the gateway application forwards the download request to a content storage application;

[0196] Step 1005, the content storage application queries a hash value of a file corresponding to an object identifier from a database;

[0197] Step 1006, the database returns the hash value;

[0198] Step 1007, the content storage application sends a file download request to a data sharing application;

[0199] Step 1008, the data sharing application returns the file to the content storage application;

[0200] Step 1009, the content storage application returns the file to the gateway application;

[0201] Step 1010, the gateway application sends a decryption request to an encryption and decryption application;

[0202] Step 1011, the encryption and decryption application returns the plaintext of the file to the gateway application;

[0203] Step 1012, the gateway application returns the plaintext of the file to the user equipment.

[0204] The data migration process is described below. Data migration is used to solve the problem of data loss caused by the offline (exit the data sharing network) of the storage node.

[0205] Referring to Figure 11 , a step flow chart of the migration method of one embodiment of the application is shown, which can specifically include the following steps:

[0206] Step 1101, the content storage application registers with the scheduling node at a regular time, and the registration period is 5 minutes;

[0207] Step 1102, the scheduling node saves the registration information of the storage node to the database;

[0208] Step 1103, the management node reads the registration information of the storage node from the database at a regular time (such as 1 hour);

[0209] Step 1104, the database returns the registration information;

[0210] Step 1105, the management node finds the offline node from the storage node;

[0211] Step 1106, the management node sends a data migration request to the record node; the data migration request can contain the information of the offline node;

[0212] Step 1107, after receiving the data migration request, the record node selects a new storage node in the same geographic region in the database; the same geographic region can refer to the same geographic location.

[0213] Step 1108, the record node merges the storage record of the offline node into the storage record of the new storage node;

[0214] Step 1109, the content storage application of the new storage node downloads the storage record from the record node at a regular time (12 hours);

[0215] Step 1110, the record node returns the storage record;

[0216] Step 1111, the content storage of the new storage node downloads the storage objects that have not been saved by using the data sharing application.

[0217] Specifically, the hash value in the storage record can be compared with the local storage, and if it is found that the local storage does not contain the file corresponding to the hash value, the data sharing application is called to download (that is, the storage objects that have not been saved are obtained from other connected storage nodes). After the files corresponding to the storage record are saved to the local storage, the data migration ends.

[0218] It should be noted that, for the sake of simplicity, the method embodiments are all described as a series of actions. However, those skilled in the art should understand that the embodiments of the present invention are not limited to the described order of actions, because according to the embodiments of the present invention, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions involved are not necessarily essential to the embodiments of the present invention.

[0219] Based on the above embodiments, this invention also provides a storage device, referring to... Figure 12 The schematic diagram shown illustrates that this storage device can store storage objects using a data sharing network. Different storage nodes in the data sharing network are connected point-to-point. This device is applied to the first storage node and may specifically include the following modules:

[0220] Storage module 1201 is used to store storage objects;

[0221] The first sending module 1202 is used to send the storage object to the connected second storage node so that the second storage node stores and / or forwards the storage object;

[0222] The second sending module 1203 is used to send the object information of the stored object to the recording node after the stored object is successfully stored, so that the recording node saves the storage record of the first storage node; the storage record includes: the object information stored in the corresponding storage node.

[0223] Optionally, the storage module 1201 may specifically include:

[0224] The upload storage module is used to store the storage object corresponding to the upload request, wherein the geographical location information corresponding to the upload request is matched with the geographical location information corresponding to the first storage node.

[0225] Optionally, the storage module 1201 may specifically include:

[0226] The determination module is used to determine the hash value corresponding to the stored object;

[0227] The storage module is used to store the mapping relationship between the object identifier and the hash value of the stored object;

[0228] The hash storage module is used to store the storage object according to the hash value.

[0229] On the basis of the above-mentioned embodiments, the embodiments of the present application further provide a migration device, which can store storage objects by using a data sharing network, different storage nodes in the data sharing network are connected point-to-point, referring to Figure 13 the structural schematic diagram shown in the figure, the migration device can include a record node 1301 and a fourth storage node 1302, wherein the record node 1301 can include:

[0230] a migration request receiving module 1311, configured to receive a data migration request;

[0231] a migration node determining module 1312, configured to determine a third storage node and a fourth storage node according to the data migration request;

[0232] a migration module 1313, configured to update storage records of the third storage node to storage records of the fourth storage node; the storage records include object information stored by corresponding storage nodes.

[0233] The fourth storage node 1302 can include:

[0234] a record obtaining module 1321, configured to obtain storage records from a record node; the storage records include object information stored by corresponding storage nodes;

[0235] a storage object obtaining module 1322, configured to obtain a first storage object from a fifth storage node if a first storage object corresponding to first object information in the storage records is not stored locally; the fourth storage node and the fifth storage node are connected point-to-point

[0236] Optionally, the data migration request includes an identifier of the third storage node, and geographical position information corresponding to the third storage node matches geographical position information corresponding to the fourth storage node.

[0237] On the basis of the above-mentioned embodiments, the embodiments of the present application further provide a download device, which can store storage objects by using a data sharing network, different storage nodes in the data sharing network are connected point-to-point, the download device can be applied to a sixth storage node, referring to Figure 14 , the device specifically can include the following modules:

[0238] a receiving module 1401, configured to receive a download request; the download request includes second object information;

[0239] an obtaining module 1402, configured to obtain a second storage object from a seventh storage node if a second storage object corresponding to the second object information is not stored locally; the sixth storage node and the seventh storage node are connected point-to-point.

[0240] Optionally, the sixth storage node can be obtained based on the second object information and the storage records of the sixth storage node.

[0241] This invention also provides an electronic device that can perform the functions of the aforementioned storage node, scheduling node, recording node, or management node.

[0242] like Figure 15 As shown, the electronic device may include a processor 1501, a communication interface 1502, a memory 1503, and a communication bus 1504, wherein the processor 1501, the communication interface 1502, and the memory 1503 communicate with each other through the communication bus 1504.

[0243] Memory 1503 is used to store computer programs;

[0244] When processor 1501 executes the program stored in memory 1503, it performs the following steps:

[0245] Store the storage object;

[0246] The storage object is sent to the connected second storage node so that the second storage node stores and / or forwards the storage object;

[0247] After successfully storing the object, the object information of the stored object is sent to the recording node so that the recording node saves the storage record of the first storage node; the storage record includes: the object information stored in the corresponding storage node.

[0248] The communication bus mentioned in the above electronic devices can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used to represent it in the diagram, but this does not indicate that there is only one bus or one type of bus.

[0249] The communication interface is used for communication between the aforementioned electronic devices and other devices.

[0250] The memory can include a random access memory (RAM) and can also include a non-volatile memory such as at least one disk memory. Optionally, the memory can also be at least one storage device located remotely from the aforementioned processor.

[0251] The processor described above can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc., and can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component.

[0252] In yet another embodiment provided by the present application, a computer readable storage medium is also provided, which stores instructions, when running on a computer, causes the computer to perform the storage method or the migration method or the download method of any of the above embodiments.

[0253] In yet another embodiment provided by the present application, a computer program product containing instructions is also provided, which, when running on a computer, causes the computer to perform the storage method or the migration method or the download method of any of the above embodiments.

[0254] In the embodiments described above, all or some of the steps can be implemented by hardware, software, firmware or any combination thereof. When implemented by software, all or some of the steps can be implemented in the form of one or more computer programs or program elements. The computer programs reside (at least temporarily) in a memory of a computer during execution. The memory can be a RAM memory, a flash memory, a ROM memory, an EPROM memory, or any other suitable memory. The memory can be integral to or separate from the computer. The computer programs can be written in any suitable programming language, such as C, C++, Java, Visual Basic, etc. The computer programs can be written in assembly or machine language, if desired. The computer programs can be distributed over network coupled file servers, or can be distributed by any other suitable means.

[0255] It is to be understood that the terminology "first", "second", etc. used herein merely for the purpose of distinguishing one entity or action from another, and does not necessarily imply these entities or actions are mutually exclusive or are either temporal or spatial priorities of one another. Moreover, the terms "comprising", "including", or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but can include other elements not expressly listed or inherent to such process, method, article, or apparatus. The term "consisting of" is intended to cover only those elements listed in the specification.

[0256] Each of the embodiments described in the specification adopt a relevant manner for description, and the same or similar parts among the embodiments can be referred to each other. Each embodiment focuses on the difference from other embodiments. In particular, for the system embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the part of the method embodiments.

[0257] The above merely provides the preferred embodiments of the application, and not intended to limit the protection scope of the application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the application shall fall within the protection scope of the application.

Claims

1. A storage method characterized by comprising: A data sharing network is used to store a storage object, different storage nodes in the data sharing network are connected in a point-to-point manner, the method is applied to a first storage node in the data sharing network, and the method comprises the following steps: In the case of receiving an upload request, a storage node is randomly specified as the first storage node, or a storage node with matched geographical position information corresponding to the upload request is specified as the first storage node, and the storage object is stored; the storage nodes connected with the first storage node are at least one; According to a backup threshold M, a second storage node is selected from the storage nodes connected with the first storage node, the number of the second storage nodes is less than or equal to the backup threshold M, and M is a positive integer; The storage object is sent to the second storage node, so that the second storage node judges whether to store and / or forward the storage object according to a comparison result of a remaining capacity of a storage space and a preset capacity; After the storage object is successfully stored, object information of the storage object is sent to a record node, so that the record node saves a storage record of the first storage node; the storage record comprises object information stored by a corresponding storage node; the object information comprises an object name or a hash value.

2. The method of claim 1, wherein, The storage object is stored, which comprises storing a storage object corresponding to the upload request, and the geographical position information corresponding to the upload request is matched with the geographical position information corresponding to the first storage node.

3. The method of claim 1, wherein, The storage object is stored, which comprises: A hash value corresponding to the storage object is determined; A mapping relationship between an object identifier of the storage object and the hash value is saved; The storage object is stored according to the hash value.

4. A migration method characterized by, A data sharing network is used to store a storage object, different storage nodes in the data sharing network are connected in a point-to-point manner, and the method comprises the following steps: A record node receives a data migration request; The record node determines a third storage node and a fourth storage node in the data sharing network according to the data migration request; the third storage node is a storage node representing a fault or an offline storage node; the fourth storage node is a storage node randomly specified in an online state or a storage node with matched geographical position information corresponding to the third storage node; The record node updates a storage record of the third storage node to a storage record of the fourth storage node; the storage record comprises object information stored by a corresponding storage node; the object information comprises an object name or a hash value; The fourth storage node obtains the storage record from the record node; the storage record comprises object information stored by a corresponding storage node; The fourth storage node acquires the first storage object from a fifth storage node in the data sharing network in a case that the first storage object corresponding to the first object information in the storage record is not stored locally; the fifth storage node is a storage node randomly specified in a case that an upload request is received, or a storage node in which geographical position information corresponding to the upload request is matched successfully; the fourth storage node is selected from storage nodes connected to the fifth storage node according to a backup threshold M, and the number of the fourth storage nodes is less than or equal to the backup threshold M, and M is a positive integer.

5. The method of claim 4, wherein, The data migration request includes an identification of a third storage node, and geographical position information corresponding to the third storage node matches the geographical position information corresponding to the fourth storage node.

6. A download method characterized by comprising: The data sharing network is used to store storage objects, different storage nodes in the data sharing network are connected point-to-point, and the method is applied to a sixth storage node and includes the following steps: A download request is received, and the download request includes second object information, and the second object information includes a second object name or a second hash value. If a second storage object corresponding to the second object information is not stored locally, the second storage object is acquired from a seventh storage node, the seventh storage node is a storage node randomly specified in a case that an upload request is received, or a storage node in which geographical position information corresponding to the upload request is matched successfully, and the sixth storage node is selected from storage nodes connected to the seventh storage node according to a backup threshold M, and the number of the sixth storage nodes is less than or equal to the backup threshold M, and M is a positive integer. The sixth storage node is a storage node in which geographical position information corresponding to the download request is matched successfully with geographical position information corresponding to the storage node.

7. The method of claim 6, wherein, The sixth storage node can also be obtained according to the second object information and a storage record of the sixth storage node.

8. A storage system, characterized by The system includes a record node and a data sharing network. The data sharing network includes a plurality of storage nodes, and different storage nodes are connected point-to-point. The record node includes: A storage record of the storage node is saved by the saving module, and the storage record includes object information stored by a corresponding storage node, and the object information includes an object name or a hash value. A data migration request is received by the migration request receiving module. The third storage node is a storage node representing a failed storage node or an offline storage node, and the fourth storage node is a storage node randomly specified in an online state or a storage node in which geographical position information corresponding to the third storage node is matched successfully. The storage record of the third storage node is updated to the storage record of the fourth storage node by the migration module, and the storage record includes object information stored by a corresponding storage node. The fourth storage node is located in the data sharing network, and is configured to obtain a storage record from the record node, and obtain a first storage object corresponding to first object information in the storage record from a fifth storage node in the data sharing network if the first storage object is not stored locally; the fifth storage node is a storage node randomly specified in a case where an upload request is received, or a storage node whose geographical position information matches the upload request; the fourth storage node is selected from storage nodes connected to the fifth storage node according to a backup threshold M, and the number of the fourth storage nodes is less than or equal to the backup threshold M, and M is a positive integer.

9. The system of claim 8, wherein, The data sharing network comprises a first storage node. The first storage node comprises: a storage module configured to store a storage object; a first sending module configured to send the storage object to a connected second storage node, so that the second storage node stores and / or forwards the storage object; a second sending module configured to send object information of the storage object to a record node after successfully storing the storage object, so that the record node saves a storage record of the first storage node; the storage record comprises object information stored by a corresponding storage node.

10. The system of claim 8, wherein, The data sharing network comprises a sixth storage node. The sixth storage node comprises: a download request receiving module configured to receive a download request; the download request comprises second object information; a storage object obtaining module configured to obtain a second storage object corresponding to the second object information from a seventh storage node if the second storage object is not stored locally.

11. An electronic device, comprising: The apparatus comprises a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory are in communication with each other through the communication bus; The memory is configured to store a computer program; The processor is configured to execute the program stored in the memory, and implement the method steps in any one of claims 1-7.

12. A computer readable storage medium having stored thereon a computer program, characterized in that, The program is executed by the processor to implement the method in any one of claims 1-7.

Images