Data sharing method, data sharing platform and device

Abstract

The invention discloses a data sharing method and device and a data sharing platform. According to the technical scheme provided by the invention, in response to a data storage demand of a user, the target data is stored in the storage cluster; performing fault detection and storage consistency detection on the storage task to obtain a corresponding detection result; according to the fault detection result and the storage consistency detection result, the storage units and / or nodes which have faults and / or do not pass the storage consistency detection are / is moved out of the storage cluster and are isolated and repaired; normal parts in the cluster are used for data storage, and after storage is completed, units and nodes with faults removed are automatically added into the storage cluster again. Multi-copy storage is carried out based on a fully-distributed redundant architecture, the efficiency and safety of data reading and storage are greatly improved, the influence of abnormal conditions on the front end is avoided through a fault hard disk isolation and recovery mechanism, customers are not aware, and the user experience is effectively improved.

Description

Technical Field

[0001] The present invention relates to the field of data storage, and in particular to a data sharing method, device and a data sharing platform. Background Art

[0002] With the rapid development of information technology, as society enters the era of big data, people are increasingly aware of the value of data, and are able to obtain incremental value through data by integrating and mining data information. In order to share the collected data, it is necessary to build a data sharing platform based on a large database.

[0003] In order to complete data sharing, the most important thing is data storage and reading. However, at present, data sharing platforms usually use cloud servers to store the information that needs to be shared, but the cloud servers used are not completely trustworthy entities. When storing data in them, problems such as storage location failure and data loss may occur during the storage or reading process. Data security cannot be guaranteed, and the storage efficiency is not high. When repairing the storage location, data loss may also occur, which affects the user's storage and reading process of other normal data at that location, resulting in interruption of the data transmission process. This greatly reduces the user's trust in the data sharing process and also reduces the user experience. Summary of the invention

[0004] In order to solve the above technical problems, the present invention provides a data sharing method and a corresponding data sharing device, computing equipment and computer storage medium.

[0005] According to one aspect of the present invention, a data sharing method is provided, the method comprising:

[0006] In response to the user's data storage requirements, the target data is stored in the storage cluster;

[0007] Perform fault detection and storage consistency detection on storage tasks, and obtain fault detection results and storage consistency detection results;

[0008] According to the fault detection results and storage consistency test results, the storage units and / or storage nodes that have failed and / or failed the storage consistency test are removed from the storage cluster and isolated, and the faults are repaired;

[0009] The normal storage function part of the storage cluster is used for data storage, and after the data storage is completed, the storage unit and / or storage node whose failure is rectified is automatically added back to the storage cluster.

[0010] In the above solution, the storage cluster includes multiple servers and / or racks; each server and / or rack serves as a storage node, and multiple storage units are set in each storage node;

[0011] The step of storing the target data in the storage cluster in response to the user's data storage demand further includes:

[0012] Based on strong consistency replication technology, the target data is replicated to generate the original target data and a preset number of copies of the target data as each item in the stored data;

[0013] The target data original and a preset number of target data copies are stored in different storage nodes respectively.

[0014] In the above solution, the storage consistency detection further includes:

[0015] Based on the target data, compare it with the original target data and the copy of the target data in the stored data to determine whether they are exactly the same as the target data;

[0016] If one of the stored data is identical to the target data, it is determined that the storage node and / or storage unit corresponding to the original or duplicate of the target data passes the comparison;

[0017] If one item in the stored data is not completely identical to the target data, it is determined that the storage node and / or storage unit corresponding to the original or duplicate of the target data has failed the comparison;

[0018] The comparison results of each item in the stored data are summarized as the storage consistency comparison result.

[0019] In the above scheme, according to the fault detection result and the storage consistency detection result, the storage unit and / or storage node that fails and / or fails the storage consistency detection is removed from the storage cluster and isolated, and the fault is repaired, further comprising:

[0020] According to the fault detection results and the storage consistency comparison results, the faulty storage nodes and / or faulty storage units that have failed and have not passed the comparison are screened out;

[0021] Suspend the use of the faulty storage node and / or the faulty storage unit for isolation;

[0022] The existing storage content in the faulty storage node and / or the faulty storage unit is copied, and the faulty storage node and / or the faulty storage unit is cleared after the copying is completed.

[0023] In the above solution, the method of using the part of the storage cluster with normal storage function to store data, and after completing the data storage, automatically adding the storage unit and / or storage node whose fault is relieved to the storage cluster again, further includes:

[0024] The existing storage contents in the replicated faulty storage nodes and / or faulty storage units, as well as the target data corresponding to the faulty storage nodes and / or faulty storage units, are evenly transferred and stored in normal storage nodes and / or storage units;

[0025] After the fault is rectified, the faulty storage node and / or the faulty storage unit are automatically added back to the storage cluster and the storage function is restored;

[0026] The storage data in the previously normal storage nodes and / or storage units are transferred and stored in the restored storage nodes and / or storage units, so that the data storage capacity in each storage node and / or storage unit in the storage cluster remains even.

[0027] In the above solution, the method for storing target data in the storage cluster in response to the user's data storage demand further includes:

[0028] Obtain the space occupancy and operating status of each storage node and / or storage unit in the storage cluster;

[0029] Based on the operating status of each storage node, currently unavailable storage nodes are eliminated;

[0030] Based on artificial intelligence and space occupancy, multiple storage nodes for storing the original target data and a preset number of target data copies are screened from the currently available storage nodes.

[0031] In the above scheme, the method based on artificial intelligence and space occupancy is used to select multiple storage nodes for storing the original target data and a preset number of target data copies from currently available storage nodes, further comprising:

[0032] Determine the optimized available space corresponding to each storage node based on the computing power resources corresponding to the storage node.

[0033]

[0034] Among them, S i is the optimized available space corresponding to the storage node, S W is the overall storage space based on the same computing power node, m is the number of storage nodes based on the same computing power node, C i is the available computing power resources of the ith storage node.

[0035] In the above scheme, the method further includes:

[0036] In response to the user's data reading and writing requirements, determine the previously stored target data corresponding to the data reading and writing requirements; wherein the data reading and writing requirements at least include reading and / or modifying and / or deleting the target data;

[0037] According to the data reading and writing requirements, the original copy of the target data that has been stored previously is processed;

[0038] Based on the original target data, determine the storage location of the corresponding target data copy;

[0039] According to the data reading and writing requirements and the storage location of the target data copies, all target data copies are processed in the same way as the target data originals.

[0040] According to another aspect of the present invention, a data sharing device is provided, comprising: a response storage module, a detection module, an isolation and repair module, and a self-healing module; wherein:

[0041] The response storage module is used to store the target data in the storage cluster in response to the user's data storage demand;

[0042] The detection module is used to perform fault detection and storage consistency detection on the storage task, and obtain the fault detection result and the storage consistency detection result;

[0043] The isolation and repair module is used to remove the storage units and / or storage nodes that have failed and / or failed the storage consistency test from the storage cluster and isolate them, and perform fault repair according to the fault detection results and the storage consistency test results;

[0044] The self-healing module uses the part of the storage cluster with normal storage function to store data, and after completing the data storage, automatically adds the storage unit and / or storage node whose fault is removed to the storage cluster again.

[0045] According to another aspect of the present invention, a data sharing platform is provided, wherein the data sharing platform is used to implement any of the above data sharing methods, including: a user control unit, a data transmission unit, a core processor, a data storage unit and a content display unit; wherein:

[0046] The user operation unit is used to determine the user's storage requirement and / or the user's read / write requirement by acquiring the user's control operation, and generate a corresponding control signal;

[0047] The core processor is used to process the control signal and generate a storage signal and / or an extraction signal for corresponding data in the control signal according to the control signal;

[0048] The data storage unit is used to store and / or extract the corresponding data in the control signal and its duplicate data according to the storage signal and / or extraction signal;

[0049] The data transmission unit is used to transmit the control signal, the corresponding data in the control signal and the copy data thereof;

[0050] The content display unit is used to display the characteristic information in the control signal and the data content extracted based on the extraction operation.

[0051] According to the technical solution provided by the present invention, in response to the user's data storage needs, the target data is stored in the storage cluster; fault detection and storage consistency detection are performed on the storage task to obtain the fault detection results and the storage consistency detection results; based on the fault detection results and the storage consistency detection results, the storage units and / or storage nodes that have failed and / or failed the storage consistency detection are removed from the storage cluster and isolated, and the fault is repaired; the part of the storage cluster with normal storage functions is used for data storage, and after the data storage is completed, the storage units and / or storage nodes with the fault removed are automatically added back to the storage cluster. Based on the user's storage needs, multiple data copies are generated using strong consistency replication technology, and are stored in multiple storage nodes or storage units set up in the storage cluster. Through the fully distributed redundant architecture, the horizontal linear expansion characteristics of the storage cluster are guaranteed, and parallel reading and writing of data are realized, which greatly improves the storage capacity and storage performance of the storage cluster. It also makes it possible for the entire system to expand capacity, fail, etc., without interrupting the current data reading and writing, reducing the steps of re-reading and writing, and reducing the probability of errors. In addition, multi-copy storage greatly improves data security, so that the loss of a single copy does not affect data recovery and the normal operation of the business, effectively improving the security of the storage process and the efficiency of business operations. Through multiple; by performing fault detection and storage consistency detection during data storage, faulty or abnormal storage in the storage cluster can be quickly screened out. Storage nodes or storage units are removed from the storage cluster and isolated. Through the isolation and recovery mechanism of faulty hard disks, as well as the corresponding data self-healing and storage balancing functions, data security is improved while avoiding the impact of abnormal situations on the front end, making customers unaware and ensuring business continuity. The storage capacity of each storage unit and node is balanced, avoiding the problem of excessive storage capacity of individual hard disks affecting the overall data read and write speed after a failure, ensuring the balance of storage capacity and the overall read and write speed; in addition, according to the user's processing method for modifying and deleting the target data, the original and duplicate copies of the target data in the corresponding storage nodes and units are processed in the same way, ensuring the consistency of the target data with the corresponding storage data in each storage location, fully ensuring the data security of users when adjusting data, and improving the user experience in the process of data sharing and revision. Finally, by obtaining the operating status and space occupancy of each storage node, based on artificial intelligence, scientific judgment and selection of nodes for this storage further improve the scientificity and rationality of the storage stage selection, and further improve the scientificity and rationality of the overall working process of the storage cluster, effectively improving the operating efficiency of the data sharing process, and greatly improving the user experience.

[0052] Other features and advantages of the present invention will be described in the following description, and partly become apparent from the description, or understood by practicing the present invention. The purpose and other advantages of the present invention can be realized and obtained by the structures particularly pointed out in the written description, claims, and drawings.

[0053] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0055] Figure 1 A schematic diagram of a data sharing method according to an embodiment of the present invention is shown;

[0056] Figure 2 A schematic diagram showing a flow chart of a data storage consistency detection method according to an embodiment of the present invention;

[0057] Figure 3 A schematic diagram of a process of isolating and restoring a storage unit and / or storage node according to an embodiment of the present invention is shown;

[0058] Figure 4A A schematic diagram of storage node isolation according to an embodiment of the present invention is shown;

[0059] Figure 4B A schematic diagram of memory cell isolation according to an embodiment of the present invention is shown;

[0060] Figure 5 A schematic diagram of storage balancing self-healing when a storage node is isolated according to an embodiment of the present invention is shown;

[0061] Figure 6 A schematic diagram of automatic storage balancing after storage node isolation and recovery according to an embodiment of the present invention is shown;

[0062] Figure 7 A schematic diagram of a process of selecting a storage location based on multi-copy storage according to an embodiment of the present invention is shown;

[0063] Figure 8 A schematic flow chart of a data sharing method according to another embodiment of the present invention is shown;

[0064] Fig. 9 A structural block diagram of a data sharing device according to an embodiment of the present invention is shown;

[0065] Fig.10A schematic structural diagram of a computing device according to an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0066] The preferred embodiments of the present invention are described below in conjunction with the accompanying drawings. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present invention, and are not used to limit the present invention.

[0067] Figure 1 A schematic flow chart of a data sharing method according to an embodiment of the present invention is shown, and the method comprises the following steps:

[0068] Step S101 , storing target data in a storage cluster in response to a user's data storage demand.

[0069] Specifically, the storage cluster includes multiple servers and / or racks; each server and / or rack serves as a storage node, and multiple storage units are set in each storage node.

[0070] The storage cluster is a decentralized, fully distributed redundant architecture. By setting up multiple storage nodes, each storage node is set up with multiple storage units, so that the storage cluster has horizontal linear expansion characteristics, can read and write data in parallel, and the storage cluster can achieve dual linear growth of storage capacity and storage performance by adding new servers (storage nodes);

[0071] At the same time, the storage cluster supports online capacity expansion. Based on a fully distributed redundant architecture, the capacity expansion process does not affect the running storage or reading business, and the upper-level system does not need to interrupt data reading and writing.

[0072] Preferably, the storage unit may be a storage medium such as a hard disk, or may be a collection of multiple storage media, which is not limited here.

[0073] Specifically, storing the target data in the storage cluster in response to the user's data storage demand further includes:

[0074] Based on strong consistency replication technology, the target data is replicated to generate the original target data and a preset number of copies of the target data as each item in the stored data;

[0075] The target data original and a preset number of target data copies are stored in different storage nodes respectively.

[0076] Preferably, the preset number is at least 2, that is, the target data is copied at least two copies of the target data through a strong consistency replication technology, and the copies are stored in different storage nodes.

[0077] Step S102, performing fault detection and storage consistency detection on the storage task, and obtaining the fault detection result and the storage consistency detection result.

[0078] Step S103: According to the fault detection result and the storage consistency detection result, the storage units and / or storage nodes that have failed and / or failed the storage consistency detection are removed from the storage cluster and isolated, and the faults are repaired.

[0079] Step S104: using the part of the storage cluster with normal storage function to store data, and after the data storage is completed, automatically adding the storage unit and / or storage node whose failure is rectified to the storage cluster again.

[0080] According to a data sharing method provided by this embodiment, in response to the user's data storage needs, the target data is stored in the storage cluster; fault detection and storage consistency detection are performed on the storage task to obtain the fault detection result and the storage consistency detection result; based on the fault detection result and the storage consistency detection result, the storage unit and / or storage node that has failed and / or failed the storage consistency detection is removed from the storage cluster and isolated, and the fault is repaired; the part of the storage cluster with normal storage function is used for data storage, and after the data storage is completed, the storage unit and / or storage node whose fault is rectified is automatically added back to the storage cluster. A data sharing method provided by this embodiment generates multiple data copies based on the user's storage needs by using strong consistency replication technology, and stores them in multiple storage nodes or storage units set in the storage cluster respectively. Through the fully distributed redundant architecture, the horizontal linear expansion characteristics of the storage cluster are guaranteed, and parallel reading and writing of data are realized, which greatly improves the storage capacity and storage performance of the storage cluster. It also makes the entire system unnecessary to interrupt the current data reading and writing when expanding or failing, reduces the steps of re-reading and writing, and reduces the probability of errors. In addition, the security of data is greatly improved through multi-copy storage, so that the loss of a single copy does not affect the normal operation of data recovery and business, and effectively improves the security of the storage process and the efficiency of business operation. Through multiple copies, fault detection and storage consistency detection are performed during the data storage process, the storage nodes or storage units with faults or abnormalities in the storage cluster can be quickly screened out, and they can be removed from the storage cluster and isolated. Through the fault hard disk isolation and recovery mechanism, while improving data security, the impact of abnormal situations on the front end is avoided, so that customers are not aware of it.

[0081] Figure 2 FIG. 4 is a flow chart showing a method for detecting consistency of data storage according to an embodiment of the present invention. Figure 2 As shown, the method comprises the following steps:

[0082] Step S201, obtaining target data and the corresponding target data original and target data copy.

[0083] Step S202, determining whether the original copy of the target data and the copy of the target data are completely identical to the target data.

[0084] Specifically, based on the target data, the target data is compared with the original target data and the copy of the target data in the stored data to determine whether they are exactly the same as the target data.

[0085] If so, that is, one item in the stored data is completely identical to the target data, then step S203 is executed; if not, that is, one item in the stored data is not completely identical to the target data, then step S204 is executed.

[0086] Step S203, determining whether the storage nodes and / or storage units corresponding to the original or duplicate copy of the target data pass the comparison.

[0087] Step S204, determining that the storage node and / or storage unit corresponding to the original or duplicate copy of the target data fails to pass the comparison.

[0088] Step S205 , summarizing the comparison results of each item in the stored data as a storage consistency comparison result.

[0089] According to the above method, by comparing the original target data with the original and the copy obtained after replication, it is possible to accurately determine whether there is a fault or storage inconsistency in the storage node or storage unit used to store the original and the copy of the target data, so as to discover problems with the storage node or unit, or problems in the stored data as early as possible, which is convenient for subsequent processing of the storage node or unit, and repair of the data, effectively improving the security of data storage and greatly reducing the impact of abnormal situations on users.

[0090] Figure 3 A schematic diagram of a process of isolating and restoring a storage unit and / or storage node according to an embodiment of the present invention is shown;

[0091] like Figure 3 As shown, the method comprises the following steps:

[0092] Step S301 : According to the fault detection result and the storage consistency comparison result, the faulty storage nodes and / or faulty storage units that have failed and have not passed the comparison are screened out.

[0093] Step S302: suspend the use of the faulty storage node and / or the faulty storage unit for isolation.

[0094] Specifically, as shown in FIG4, Figure 4AA schematic diagram of storage node isolation according to an embodiment of the present invention is shown; Figure 4B A schematic diagram of memory cell isolation according to an embodiment of the present invention is shown.

[0095] Step S303, copy the existing storage content in the faulty storage node and / or the faulty storage unit, and clear the faulty storage node and / or the faulty storage unit after the copying is completed.

[0096] Step S304, the existing storage contents in the replicated faulty storage nodes and / or faulty storage units, as well as the target data corresponding to the faulty storage nodes and / or faulty storage units, are evenly transferred and stored in normal storage nodes and / or storage units.

[0097] Specifically, Figure 5 As shown, Figure 5 A schematic diagram of storage balancing self-healing when a storage node is isolated according to an embodiment of the present invention is shown.

[0098] Step S305: After the fault is rectified, the faulty storage node and / or the faulty storage unit are automatically added back to the storage cluster, and the storage function is restored.

[0099] Step S306, transferring the storage data in the previously normal storage nodes and / or storage units to the restored storage nodes and / or storage units, so that the data storage capacity in each storage node and / or storage unit in the storage cluster remains even.

[0100] Specifically, Figure 6 As shown, Figure 6 A schematic diagram of automatic storage balancing after storage node isolation and recovery according to an embodiment of the present invention is shown.

[0101] According to the above method, through the isolation and recovery mechanism of faulty hard disks, as well as the corresponding data self-healing and storage balancing functions, while improving data security, it avoids the impact of abnormal situations on the front end, making customers unaware, ensuring business continuity, and balancing the storage capacity of each storage unit and node, avoiding the problem of excessive storage capacity of individual hard disks after a failure, affecting the overall data reading and writing speed, thereby ensuring the balance of storage capacity and the overall reading and writing speed.

[0102] Figure 7 A schematic diagram of a process of selecting a storage location based on multi-copy storage according to an embodiment of the present invention is shown;

[0103] like Figure 7 As shown, the method comprises the following steps:

[0104] Step S701, obtaining the space occupancy and operation status of each storage node and / or storage unit in the storage cluster.

[0105] Step S702: based on the operation status of each storage node, remove the currently unavailable storage nodes.

[0106] Preferably, the unavailable storage node may be a storage node whose operating status is abnormal such as a fault, or a storage node that is not currently performing storage operations based on user requirements. The specific situation can be set according to the actual needs of the user and is not limited here.

[0107] Step S703, based on artificial intelligence and space occupancy, multiple storage nodes for storing the original target data and a preset number of target data copies are screened from currently available storage nodes.

[0108] Preferably, the optimized available space corresponding to each storage node is determined according to the computing power resources corresponding to the storage node.

[0109]

[0110] Among them, S i is the optimized available space corresponding to the storage node, S W is the overall storage space based on the same computing power node, m is the number of storage nodes based on the same computing power node, C i is the available computing power resources of the ith storage node.

[0111] Preferably, the computing power node is a data processor corresponding to the storage node; the computing power resource may be the computing power status of the computing power node corresponding to the storage node.

[0112] According to the above method, by obtaining the operating status and space occupancy of each storage node, based on artificial intelligence, scientific judgment and selection of the nodes used for this storage are made, which further improves the scientificity and rationality of the storage stage selection, and also further improves the scientificity and rationality of the overall working process of the storage cluster, effectively improving the operating efficiency of the data sharing process and greatly improving the user experience.

[0113] Figure 8 A schematic flow chart of a data sharing method according to another embodiment of the present invention is shown;

[0114] like Figure 8 As shown, the method comprises the following steps:

[0115] Step S801 , in response to a user's data reading and writing demand, determining previously stored target data corresponding to the data reading and writing demand.

[0116] Specifically, the data reading and writing requirements at least include reading and / or modifying and / or deleting target data.

[0117] Step S802: Process the original copy of the target data that has been stored previously according to the data reading and writing requirements.

[0118] Step S803, based on the target data original, determine the storage location of the corresponding target data copy.

[0119] Step S804, according to the data reading and writing requirements and the storage location of the target data copies, all the target data copies are processed in the same way as the target data original.

[0120] According to the above method, the original and duplicate target data in the corresponding storage nodes and units can be processed in the same way according to the user's modification and deletion processing methods for the target data, thereby ensuring the consistency of the target data with the data stored in the corresponding storage locations, fully guaranteeing the user's data security when adjusting the data, and effectively improving the user experience of data sharing.

[0121] Fig. 9 FIG. 4 shows a structural block diagram of a data sharing device according to an embodiment of the present invention. Fig. 9 As shown, the system includes: a response storage module 901, a detection module 902, an isolation and repair module 903 and a self-healing module 904; wherein,

[0122] The response storage module 901 is used to store target data in the storage cluster in response to the user's data storage demand.

[0123] Specifically, the storage cluster includes multiple servers and / or racks; each server and / or rack serves as a storage node, and multiple storage units are set in each storage node.

[0124] Specifically, the response storage module 901 is further used to:

[0125] Based on strong consistency replication technology, the target data is replicated to generate the original target data and a preset number of copies of the target data as each item in the stored data;

[0126] The target data original and a preset number of target data copies are stored in different storage nodes respectively.

[0127] Specifically, the response storage module 901 is further used to:

[0128] Obtain the space occupancy and operating status of each storage node and / or storage unit in the storage cluster;

[0129] Based on the operating status of each storage node, currently unavailable storage nodes are eliminated;

[0130] Based on artificial intelligence and space occupancy, multiple storage nodes for storing the original target data and a preset number of target data copies are screened from the currently available storage nodes.

[0131] Specifically, the optimized available space corresponding to each storage node is determined according to the computing power resources corresponding to the storage node.

[0132]

[0133] Among them, S i is the optimized available space corresponding to the storage node, S W is the overall storage space based on the same computing power node, m is the number of storage nodes based on the same computing power node, C i is the available computing power resources of the ith storage node.

[0134] The detection module 902 is used to perform fault detection and storage consistency detection on the storage task, and obtain the fault detection result and the storage consistency detection result.

[0135] Specifically, the detection module 902 is further used to:

[0136] Based on the target data, compare it with the original target data and the copy of the target data in the stored data to determine whether they are exactly the same as the target data;

[0137] If one of the stored data is identical to the target data, it is determined that the storage node and / or storage unit corresponding to the original or duplicate of the target data passes the comparison;

[0138] If one item in the stored data is not completely identical to the target data, it is determined that the storage node and / or storage unit corresponding to the original or duplicate of the target data has failed the comparison;

[0139] The comparison results of each item in the stored data are summarized as the storage consistency comparison result.

[0140] The isolation and repair module 903 is used to remove storage units and / or storage nodes that have failed and / or failed the storage consistency test from the storage cluster and isolate them, and perform fault repair according to the fault detection result and the storage consistency test result.

[0141] Specifically, the isolation repair module 903 is further used to:

[0142] According to the fault detection results and the storage consistency comparison results, the faulty storage nodes and / or faulty storage units that have failed and have not passed the comparison are screened out;

[0143] Suspend the use of the faulty storage node and / or the faulty storage unit for isolation;

[0144] The existing storage content in the faulty storage node and / or the faulty storage unit is copied, and the faulty storage node and / or the faulty storage unit is cleared after the copying is completed.

[0145] The self-healing module 904 uses the part of the storage cluster with normal storage functions to store data, and after completing the data storage, automatically adds the storage units and / or storage nodes whose failures have been resolved back to the storage cluster.

[0146] Specifically, the self-healing module 904 is further used to:

[0147] The existing storage contents in the replicated faulty storage nodes and / or faulty storage units, as well as the target data corresponding to the faulty storage nodes and / or faulty storage units, are evenly transferred and stored in normal storage nodes and / or storage units;

[0148] After the fault is rectified, the faulty storage node and / or the faulty storage unit are automatically added back to the storage cluster and the storage function is restored;

[0149] The storage data in the previously normal storage nodes and / or storage units are transferred and stored in the restored storage nodes and / or storage units, so that the data storage capacity in each storage node and / or storage unit in the storage cluster remains even.

[0150] The data sharing device further includes: a data modification module 905;

[0151] The data modification module 905 is used to determine the previously stored target data corresponding to the data reading and writing requirements in response to the user's data reading and writing requirements; wherein the data reading and writing requirements at least include reading and / or modifying and / or deleting the target data;

[0152] According to the data reading and writing requirements, the original copy of the target data that has been stored previously is processed;

[0153] Based on the original target data, determine the storage location of the corresponding target data copy;

[0154] According to the data reading and writing requirements and the storage location of the target data copies, all target data copies are processed in the same way as the target data originals.

[0155] A data sharing device provided according to this embodiment includes: a response storage module, a detection module, an isolation and repair module, and a self-healing module; wherein the response storage module is used to store target data in a storage cluster in response to a user's data storage demand; the detection module is used to perform fault detection and storage consistency detection on the storage task, and obtain the fault detection result and the storage consistency detection result; the isolation and repair module is used to remove the storage unit and / or storage node that has failed and / or failed the storage consistency detection from the storage cluster and isolate it according to the fault detection result and the storage consistency detection result, and perform fault repair; the self-healing module uses the part of the storage cluster with normal storage function to store data, and after the data storage is completed, automatically adds the storage unit and / or storage node whose fault has been resolved back to the storage cluster. A data sharing device provided by the present embodiment generates multiple data copies by using strong consistency replication technology based on the storage needs of users, and stores them in multiple storage nodes or storage units set in the storage cluster respectively. Through the fully distributed redundant architecture, the horizontal linear expansion characteristic of the storage cluster is guaranteed, and parallel reading and writing of data is realized, which greatly improves the storage capacity and storage performance of the storage cluster. It also makes the entire system do not need to interrupt the current data reading and writing when expanding or encountering failures, reduces the steps of re-reading and writing, and reduces the probability of errors. In addition, the security of data is greatly improved through multi-copy storage, so that the loss of a single copy does not affect data recovery and the normal operation of the business, effectively improving the security of the storage process and the efficiency of business operation. By performing fault detection and storage consistency detection during the data storage process, the storage cluster can be quickly screened out. There are faulty or abnormal storage nodes or storage units, which are removed from the storage cluster and isolated. Through the isolation and recovery mechanism of faulty hard disks, as well as the corresponding data self-healing and storage balancing functions, data security is improved while avoiding the impact of abnormal situations on the front end, making customers unaware and ensuring business continuity. The storage capacity of each storage unit and node is balanced to avoid the problem of excessive storage capacity of individual hard disks affecting the overall data read and write speed after a failure, ensuring the balance of storage capacity and the overall read and write speed. In addition, according to the user's modification and deletion of target data, the original and copy of the target data in the corresponding storage nodes and units are processed in the same way, ensuring the consistency of the target data with the corresponding storage location The data stored in the data is guaranteed, fully guaranteeing the user's data security when data is adjusted, and improving the user experience in the process of data sharing and revision.Finally, by obtaining the operating status and space occupancy of each storage node, based on artificial intelligence, scientific judgment and selection of the nodes used for this storage are made, which further improves the scientificity and rationality of the storage stage selection, and also further improves the scientificity and rationality of the overall working process of the storage cluster, effectively improving the operating efficiency of the data sharing process and greatly improving the user experience.

[0156] The present invention also provides a data sharing platform, which is used to implement any of the above data sharing methods, including: a user control unit, a data transmission unit, a core processor, a data storage unit and a content display unit; wherein,

[0157] The user operation unit is used to determine the user's storage demand and / or the user's reading and writing demand by acquiring the user's control operation, and generate a corresponding control signal;

[0158] The core processor is used to process the control signal and generate a storage signal and / or an extraction signal for corresponding data in the control signal according to the control signal;

[0159] The data storage unit is used to store and / or extract the corresponding data in the control signal and its duplicate data according to the storage signal and / or extraction signal;

[0160] The data transmission unit is used to transmit the control signal, the corresponding data in the control signal and the copy data thereof;

[0161] The content display unit is used to display the characteristic information in the control signal and the data content extracted based on the extraction operation.

[0162] The present invention also provides a non-volatile computer storage medium, which stores at least one executable instruction, and the executable instruction can execute a data sharing method in any of the above method embodiments.

[0163] Fig.10 A schematic diagram of the structure of a computing device according to an embodiment of the present invention is shown. The specific embodiment of the present invention does not limit the specific implementation of the computing device.

[0164] like Fig.10 As shown, the computing device may include: a processor (processor) 1002 , a communication interface (Communications Interface) 1004 , a memory (memory) 1006 , and a communication bus 1008 .

[0165] in:

[0166] The processor 1002 , the communication interface 1004 , and the memory 1006 communicate with each other via a communication bus 1008 .

[0167] The communication interface 1004 is used to communicate with other devices such as clients or other servers.

[0168] The processor 1002 is used to execute the program 1010, and specifically can execute the relevant steps in the above data sharing method embodiment.

[0169] Specifically, the program 1010 may include program codes, and the program codes include computer operation instructions.

[0170] The processor 1002 may be a central processing unit (CPU), or an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present invention. The one or more processors included in the computing device may be processors of the same type, such as one or more CPUs; or processors of different types, such as one or more CPUs and one or more ASICs.

[0171] The memory 1006 is used to store the program 1010. The memory 1006 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0172] Program 1010 can be specifically used to enable processor 1002 to execute a data sharing method in any of the above method embodiments. The specific implementation of each step in program 1010 can refer to the corresponding description in the corresponding steps and units in the above data sharing method embodiment, which will not be repeated here. Those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working process of the above-described devices and modules can refer to the corresponding process description in the above-mentioned method embodiment, which will not be repeated here.

[0173] The algorithm and display provided herein are not inherently related to any particular computer, virtual system or other device. Various general purpose systems can also be used together with the teachings based on this. According to the above description, it is obvious that the structure required for constructing such systems. In addition, the present invention is not directed to any specific programming language either. It should be understood that various programming languages ​​can be utilized to realize the content of the present invention described herein, and the description of the above specific languages ​​is for disclosing the best mode of the present invention.

[0174] In the description provided herein, a large number of specific details are described. However, it is understood that embodiments of the present invention can be practiced without these specific details. In some instances, well-known methods, structures and techniques are not shown in detail so as not to obscure the understanding of this description.

[0175] Similarly, it should be understood that in order to streamline the present disclosure and aid in understanding one or more of the various inventive aspects, in the above description of exemplary embodiments of the present invention, various features of the present invention are sometimes grouped together into a single embodiment, figure, or description thereof. However, this disclosed method should not be interpreted as reflecting the intention that the claimed invention requires more features than those expressly recited in each claim. Rather, as reflected in the claims, inventive aspects lie in less than all of the features of the individual embodiments previously disclosed. Therefore, the claims that follow the detailed description are hereby expressly incorporated into the detailed description, with each claim itself serving as a separate embodiment of the present invention.

[0176] Those skilled in the art will appreciate that the modules in the devices in the embodiments may be adaptively changed and arranged in one or more devices different from the embodiments. The modules or units or components in the embodiments may be combined into one module or unit or component, and in addition they may be divided into a plurality of submodules or subunits or subcomponents. Except that at least some of such features and / or processes or units are mutually exclusive, all features disclosed in this specification (including the accompanying claims, abstracts and drawings) and all processes or units of any method or device disclosed in this manner may be combined in any combination. Unless otherwise expressly stated, each feature disclosed in this specification (including the accompanying claims, abstracts and drawings) may be replaced by an alternative feature providing the same, equivalent or similar purpose.

[0177] In addition, those skilled in the art will appreciate that, although some embodiments described herein include certain features included in other embodiments but not other features, the combination of features of different embodiments is meant to be within the scope of the present invention and form different embodiments. For example, in the claims, any one of the claimed embodiments may be used in any combination.

[0178] The various component embodiments of the present invention can be implemented in hardware, or in software modules running on one or more processors, or in a combination thereof. It should be understood by those skilled in the art that a microprocessor or digital signal processor (DSP) can be used in practice to implement some or all of the functions of some or all of the components in the embodiments of the present invention. The present invention can also be implemented as a device or apparatus program (e.g., computer program and computer program product) for executing part or all of the methods described herein. Such a program implementing the present invention can be stored on a computer-readable medium, or can have the form of one or more signals. Such a signal can be downloaded from an Internet website, or provided on a carrier signal, or provided in any other form.

[0179] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A data sharing method, comprising: In response to the user's data storage requirements, the target data is stored in the storage cluster; Perform fault detection and storage consistency detection on storage tasks, and obtain fault detection results and storage consistency detection results; According to the fault detection results and storage consistency test results, the storage units and / or storage nodes that have failed and / or failed the storage consistency test are removed from the storage cluster and isolated, and the faults are repaired; The normal storage function part of the storage cluster is used for data storage, and after the data storage is completed, the storage unit and / or storage node whose failure is rectified is automatically added back to the storage cluster.

2. A data sharing method according to claim 1, characterized in that: The storage cluster includes multiple servers and / or racks; each server and / or rack serves as a storage node, and multiple storage units are set in each storage node; The step of storing the target data in the storage cluster in response to the user's data storage demand further includes: Based on strong consistency replication technology, the target data is replicated to generate the original target data and a preset number of copies of the target data as each item in the stored data; The target data original and a preset number of target data copies are stored in different storage nodes respectively.

3. A data sharing method according to claim 1, characterized in that: The storage consistency detection further includes: Based on the target data, compare it with the original target data and the copy of the target data in the stored data to determine whether they are exactly the same as the target data; If one of the stored data is identical to the target data, it is determined that the storage node and / or storage unit corresponding to the original or duplicate of the target data passes the comparison; If one item in the stored data is not completely identical to the target data, it is determined that the storage node and / or storage unit corresponding to the original or duplicate of the target data has failed the comparison; The comparison results of each item in the stored data are summarized as the storage consistency comparison result.

4. A data sharing method according to claim 1, characterized in that: The method further includes: removing the storage unit and / or storage node that has failed and / or failed the storage consistency test from the storage cluster and isolating the storage unit and / or storage node according to the fault detection result and the storage consistency test result, and performing fault repair. According to the fault detection results and the storage consistency comparison results, the faulty storage nodes and / or faulty storage units that have failed and have not passed the comparison are screened out; Suspend the use of the faulty storage node and / or the faulty storage unit for isolation; The existing storage content in the faulty storage node and / or the faulty storage unit is copied, and the faulty storage node and / or the faulty storage unit is cleared after the copying is completed.

5. A data sharing method according to claim 1, characterized in that: The method of using the part of the storage cluster with normal storage function to store data, and automatically adding the storage unit and / or storage node whose failure is relieved to the storage cluster after the data storage is completed, further includes: The existing storage contents in the replicated faulty storage nodes and / or faulty storage units, as well as the target data corresponding to the faulty storage nodes and / or faulty storage units, are evenly transferred and stored in normal storage nodes and / or storage units; After the fault is rectified, the faulty storage node and / or the faulty storage unit are automatically added back to the storage cluster and the storage function is restored; The storage data in the previously normal storage nodes and / or storage units are transferred and stored in the restored storage nodes and / or storage units, so that the data storage capacity in each storage node and / or storage unit in the storage cluster remains even.

6. A data sharing method according to claim 1, characterized in that: The method for storing target data in a storage cluster in response to a user's data storage demand further includes: Obtain the space occupancy and operating status of each storage node and / or storage unit in the storage cluster; Based on the operating status of each storage node, currently unavailable storage nodes are eliminated; Based on artificial intelligence and space occupancy, multiple storage nodes for storing the original target data and a preset number of target data copies are screened from the currently available storage nodes.

7. A data sharing method according to claim 6, characterized in that: The method of screening multiple storage nodes for storing the original target data and a preset number of target data copies from currently available storage nodes based on artificial intelligence and space occupancy further includes: Determine the optimized available space corresponding to each storage node based on the computing power resources corresponding to the storage node. Among them, S i is the optimized available space corresponding to the storage node, S W is the overall storage space based on the same computing power node, m is the number of storage nodes based on the same computing power node, C i is the available computing power resources of the ith storage node.

8. A data sharing method according to claim 1, characterized in that: The method further comprises, In response to the user's data reading and writing requirements, determine the previously stored target data corresponding to the data reading and writing requirements; wherein the data reading and writing requirements at least include reading and / or modifying and / or deleting the target data; According to the data reading and writing requirements, the original copy of the target data that has been stored previously is processed; Based on the original target data, determine the storage location of the corresponding target data copy; According to the data reading and writing requirements and the storage location of the target data copies, all target data copies are processed in the same way as the target data originals.

9. A data sharing device, comprising: Response storage module, detection module, isolation repair module and self-healing module; wherein, The response storage module is used to store the target data in the storage cluster in response to the user's data storage demand; The detection module is used to perform fault detection and storage consistency detection on the storage task, and obtain the fault detection result and the storage consistency detection result; The isolation and repair module is used to remove the storage units and / or storage nodes that have failed and / or failed the storage consistency test from the storage cluster and isolate them, and perform fault repair according to the fault detection results and the storage consistency test results; The self-healing module uses the part of the storage cluster with normal storage function to store data, and after completing the data storage, automatically adds the storage unit and / or storage node whose fault is relieved to the storage cluster again.

10. A data sharing platform, wherein the data sharing platform is used to implement a data sharing method according to any one of claims 1 to 8, comprising: User control unit, data transmission unit, core processor, data storage unit and content display unit; wherein, The user operation unit is used to determine the user's storage requirement and / or the user's read / write requirement by acquiring the user's control operation, and generate a corresponding control signal; The core processor is used to process the control signal and generate a storage signal and / or an extraction signal for corresponding data in the control signal according to the control signal; The data storage unit is used to store and / or extract the corresponding data in the control signal and its duplicate data according to the storage signal and / or extraction signal; The data transmission unit is used to transmit the control signal, the corresponding data in the control signal and the copy data thereof; The content display unit is used to display the characteristic information in the control signal and the data content extracted based on the extraction operation.

Images