Storage optimization method and system

Abstract

The invention discloses a storage optimization method and a storage optimization system. The storage optimization method comprises the following steps: acquiring data needing to be stored; the storage priority of the obtained data is determined; establishing a data storage mechanism according to the storage priority; initiating a storage request, confirming a data storage position according to a data storage mechanism, and completing data storage. Compared with the prior art, the method and the device have the advantages that the execution time can be saved for the subsequent operation by judging from the source before the data is stored and when the data is acquired, and different storage mechanisms are provided according to different data in the subsequent storage process, so that the data storage time is prolonged, and the data storage accuracy is improved.

Description

Technical Field

[0001] This application relates to the field of data processing, and specifically, to a storage optimization method and its system. Background Art

[0002] The characteristics and storage requirements of big data: Massive, high-speed, diverse, authentic, and complex constitute the five characteristics of big data. With the continuous improvement of the technological level, using big data to achieve data storage has become increasingly popular, but it also poses higher requirements for data storage. At present, most computers simply judge by data capacity and then store data, and more and more loopholes have emerged. For example, if the computer or data has an abnormality during the data storage process, the storage process will be directly interrupted. This is not a rigorous storage method that can improve the data storage time and success rate. If other hardware devices are added to assist in data storage, it will increase the cost and cannot meet the diversification of storage.

[0003] Therefore, how to provide an optimized data storage method to improve the data storage success rate while also saving storage time has become an urgent problem to be solved in this field. Summary of the Invention

[0004] This application proposes a storage optimization method, including the following steps: obtaining the data to be stored; determining the storage priority of the obtained data; establishing a data storage mechanism according to the storage priority; initiating a storage request, confirming the data storage location according to the data storage mechanism, and completing the data storage.

[0005] As above, among them, determining the storage priority of the obtained data includes the following sub-steps: performing data anomaly detection; determining the storage priority according to the anomaly detection result.

[0006] As above, among them, performing data anomaly detection includes the following sub-steps: judging whether there is an error in the obtained data; if there is an error in the obtained data, the process exits; if the obtained data is normal, then judge whether there is an error in the data itself.

[0007] As above, among them, if the data itself is normal, the anomaly detection result is normal; if the data itself is abnormal, the anomaly detection result is abnormal; setting the data with a normal detection result as the first storage priority, and setting the data with an abnormal anomaly detection result as the second storage priority.

[0008] As above, among them, establishing a data storage mechanism according to the storage priority includes the following sub-steps: establishing a first storage mechanism according to the first storage priority; establishing a second storage mechanism according to the second storage priority.

[0009] A storage optimization system specifically includes: an acquisition unit, a priority determination unit, a storage mechanism establishment unit, and a storage unit; the acquisition unit is used to acquire data to be stored; the priority determination unit is used to determine the storage priority of the acquired data; the storage mechanism establishment unit is used to establish a data storage mechanism according to the storage priority; the storage unit is used to initiate a storage request, confirm the data storage location according to the data storage mechanism, and complete data storage.

[0010] As described above, among them, the priority determination unit's determination of the storage priority of the acquired data includes the following sub-steps: performing data anomaly detection; determining the storage priority according to the anomaly detection result.

[0011] As described above, among them, the priority determination unit's performance of data anomaly detection includes the following sub-steps: determining whether there is an error in the acquired data; if there is an error in the acquired data, the process exits; if the acquired data is normal, determining whether there is an error in the data itself.

[0012] As described above, among them, in the priority determination unit, if the data itself is normal, the anomaly detection result is normal, and if the data itself is abnormal, the anomaly detection result is abnormal; the data with a normal detection result is set as the first storage priority, and the data with an abnormal anomaly detection result is set as the second storage priority.

[0013] As described above, among them, the storage mechanism establishment unit's establishment of a data storage mechanism according to the storage priority includes the following sub-steps: establishing a first storage mechanism according to the first storage priority; establishing a second storage mechanism according to the second storage priority.

[0014] This application has the following beneficial effects:

[0015] This application can, before storing data, first perform judgment during data acquisition, make judgments from the source, save execution time for subsequent operations, and in the subsequent storage process, propose different storage mechanisms according to different data, improve the data storage time, and improve the accuracy of data storage. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the drawings in the following description are only some embodiments recorded in the present application. For those of ordinary skill in the art, other drawings can also be obtained according to these drawings.

[0017] Figure 1 is a flowchart of the storage optimization method provided according to the embodiments of the present application;

[0018] Figure 2It is a schematic internal structure diagram of a storage optimization system provided according to an embodiment of the present application. Detailed implementation manners

[0019] Next, in combination with the accompanying drawings in the embodiments of the present application, the technical solutions in the embodiments of the present application will be clearly and completely described. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative efforts belong to the scope of protection of the present application.

[0020] The present application provides a storage optimization method and its system, which can, before storing data, first perform a judgment during data acquisition, make a judgment from the source to save execution time for subsequent operations, and during the subsequent storage process, propose different storage mechanisms according to different data, improve the data storage time, and improve the accuracy of data storage.

[0021] Embodiment 1

[0022] As Figure 1 shown, this embodiment provides a storage optimization method, which specifically includes the following steps:

[0023] Step S110: Obtain the data that needs to be stored.

[0024] Among them, the data that needs to be stored can be obtained from any platform or actively input into the system.

[0025] Step S120: Determine the storage priority of the obtained data.

[0026] Among them, the amount of data that needs to be stored is huge. Therefore, this embodiment differentiates the data so as to be able to respond to data storage faster.

[0027] Before data storage, since the data is not verified, if data storage is blindly performed, it may cause waste of resources. Therefore, in this embodiment, before data storage, the data that needs to be stored also needs to be further processed, which specifically includes the following sub-steps:

[0028] Step S1201: Perform data anomaly detection.

[0029] Among them, performing data anomaly detection includes:

[0030] Step S12011: Determine whether there is an error in the obtained data.

[0031] Among them, during the process of obtaining data, due to system errors, it may also cause the failure of obtaining data, resulting in the detection of data anomalies. Therefore, in this embodiment, the detection of the obtained data is first performed.

[0032] Specifically, this embodiment determines whether there is an acquisition error at present based on the error rate of the system acquiring data in the past, wherein the error rate of the system acquiring data in the past is specifically expressed as:

[0033]

[0034] Among them, P i (x i ) indicates that data x has been successfully obtained in the past i On the probability distribution curve P i The area under (x), where the distribution curve P i (x) is the distribution curve of all data successfully obtained by the staff in the historical time period in advance, σ i Represents the distribution curve P i The standard deviation of (x).

[0035] If the error rate of the acquired data is greater than the specified threshold, it is considered that the system may be abnormal at this time, and the process will exit and further verification of the system will be carried out.

[0036] If the error rate of the acquired data is less than the specified threshold, the system is considered normal, and step S12012 is executed.

[0037] By checking whether there are any anomalies in the acquired data, anomaly detection can be started at the source. If there are any errors, it may cause anomalies in subsequent storage. Therefore, if the error rate of acquired data is less than the specified threshold, it will exit directly and no longer execute the subsequent storage process, which saves execution time and provides further guarantee for improving the success rate of storage.

[0038] Step S12012: Determine whether there is any error in the data itself.

[0039] The acquired data is compared with the original data. If they are completely consistent, it means that there is no error in the acquired data itself, and the anomaly detection result is normal. Otherwise, it is considered that there is an error in the data itself, and the anomaly detection result is abnormal.

[0040] Step S1202: Determine storage priority according to the abnormality detection result.

[0041] The data whose abnormality detection result is normal is set as the first storage priority, and the data whose abnormality detection result is abnormal is set as the second storage priority.

[0042] The storage of the first storage priority data is performed preferentially.

[0043] Step S130: Establishing a data storage mechanism according to storage priority.

[0044] Among them, establishing a data storage mechanism according to storage priorities includes the following sub-steps:

[0045] Step S1301: Establish a first storage mechanism according to the first storage priority.

[0046] In the first storage mechanism, it includes: creating a data copy and storing both the original data and the data copy simultaneously.

[0047] Since the first storage priority is for normal data, in order to ensure the accuracy of this data during subsequent transmission, a data copy is created in advance. If an error occurs during subsequent transmission, the data copy can be directly read out to ensure that subsequent operations can continue to run.

[0048] Step S1302: Establish a second storage mechanism according to the second storage priority.

[0049] In the second storage mechanism, it includes: only storing the original data.

[0050] Step S140: Initiate a storage request, confirm the data storage location according to the data storage mechanism, and complete the data storage.

[0051] Before confirming the data storage location according to the data storage mechanism, it also includes judging the storage request.

[0052] Specifically, after determining the storage mechanism, normal data and abnormal data can initiate a storage request. The storage request contains the data volume of the corresponding normal or abnormal data. Generally, there can be one or more storage requests, and each storage request can contain multiple types of data. For example, a storage request contains multiple normal data. Therefore, when there are multiple storage requests, the storage requests may or may not be processed normally by the system, so specific judgments need to be made for multiple storage requests.

[0053] If there are K storage requests, the K storage requests are respectively defined as (1, 2,... k,... K). Assume that the data volumes contained in the m-th storage request are j k1 , j k2 ... j ki ,... j kI , j ki indicating the data volume size of the i-th data contained in the k-th storage request.

[0054] Furthermore, different weights are assigned to different data volumes, where the weight represents the importance degree of this type of data.

[0055] The criteria for assigning weights are as follows: when the data volume of the data is greater than the specified threshold, a larger weight is assigned to it; when the data volume of this type of data is less than the specified threshold, a smaller weight is assigned to it, where the data volume is j k1 ,j k2 ...j ki ,...j kI The assigned weights are ω k1 ,ω k2 ,...ω ki ,...ω KI , ω1 + ω2 +...ω p +...+ω I =1, λ mn represents the weight value of the i-th data included in the k-th storage request.

[0056] The above weight assignment is pre-set by the staff, and the specific weight values are not limited here.

[0057] Among them, the processing standard U is determined according to the above parameters:

[0058]

[0059] Among them, q k represents the number of successful acquisitions of the k-th storage request, q′ k represents the number of failed acquisitions of the k-th storage request, and I represents the number of data included in the k-th storage request.

[0060] When U is greater than the specified threshold, it indicates that the system can no longer fully process multiple acquired storage requests at this time, and the process exits. Otherwise, the data storage location is confirmed.

[0061] Among them, confirming the data storage location is to confirm the node for storing data, which specifically includes the following sub-steps:

[0062] Step S1401: Determine the first storage node according to the storage request and the first storage priority, and store the normal data.

[0063] Among them, if the storage request is to store normal data, a specified number of first storage nodes are set.

[0064] Among them, in the first storage mechanism, since it is necessary to store data replicas and original data, in this embodiment, the criterion for selecting the first storage node is: the selected first storage node is a node that does not currently store data, so that more storage space can be provided for storing data replicas and original data.

[0065] Step S1402: Determine the second storage node according to the storage request and the second storage priority, and store the abnormal data.

[0066] Since the second storage mechanism only needs to store the original data, in this embodiment, the criterion for selecting the second storage node is: the selected second storage node is a node that has already partially stored other data, as long as the memory of this node can accommodate the storage of the original data.

[0067] Embodiment 2

[0068] As Figure 2 shown, this embodiment provides a storage optimization system, specifically including: an acquisition unit 201, a priority determination unit 202, a storage mechanism establishment unit 203, and a storage unit 204.

[0069] The acquisition unit is used to acquire the data that needs to be stored.

[0070] Among them, the data that needs to be stored can be acquired from any platform or actively input into the system.

[0071] The priority determination unit is used to determine the storage priority of the acquired data.

[0072] Since the data volume of the data that needs to be stored is huge, in this embodiment, the data is differentiated, so as to be able to respond to the storage of the data faster.

[0073] Before data storage, since the data is not verified, if data storage is blindly carried out, it may cause waste of resources. Therefore, in this embodiment, before data storage, the data that needs to be stored also needs to be further processed, specifically including the following sub-steps:

[0074] Step S1: Perform data anomaly detection.

[0075] Among them, performing data anomaly detection includes:

[0076] Step S11: Determine whether there is an error in the acquired data.

[0077] Among them, during the process of acquiring data, due to system errors, it may also lead to the failure of acquiring data, thus resulting in the detection result of data anomalies. Therefore, in this embodiment, the detection of the acquired data is first carried out.

[0078] Specifically, in this embodiment, it is determined whether there is an acquisition error currently according to the error rate of the system's previous data acquisition. Among them, the error rate of the previous data acquisition is specifically expressed as:

[0079]

[0080] Among them, P i (x i ) represents the previous successful acquisition of data xi Under the area of the probability distribution curve P i (x), where the distribution curve P i (x) is the distribution curve that the staff has pre-determined according to the successful acquisition of all data in the historical time period, and σ i represents the standard deviation of the distribution curve P i (x).

[0081] If the error rate of data acquisition is greater than the specified threshold, it is considered that the system may be abnormal at this time, and the process exits for further verification of the system.

[0082] If the error rate of data acquisition is less than the specified threshold, it is considered that the system is normal, and step S12 is executed.

[0083] Among them, through the verification of whether there is an abnormality in the acquired data, anomaly detection can be started at the source. If there is an error, it may cause abnormalities in subsequent storage. Therefore, if the error rate of the acquired data is less than the specified threshold, it directly exits and no longer executes the subsequent storage process, saving execution time and providing further guarantee for improving the success rate of storage.

[0084] Step S12: Determine whether there is an error in the data itself.

[0085] Among them, the acquired data is compared with the original data. If they are exactly the same, it means that there is no error in the acquired data itself, and the anomaly detection result is normal. Otherwise, it is considered that there is an error in the data itself, and the anomaly detection result is abnormal.

[0086] Step S2: Determine the storage priority according to the anomaly detection result.

[0087] Set the data with a normal anomaly detection result as the first storage priority, and set the data with an abnormal anomaly detection result as the second storage priority.

[0088] Give priority to storing the data with the first storage priority.

[0089] The storage mechanism is established to establish a data storage mechanism according to the storage priority.

[0090] Among them, establishing a data storage mechanism according to the storage priority includes the following sub-steps:

[0091] Step T1: Establish the first storage mechanism according to the first storage priority.

[0092] In the first storage mechanism, it includes: creating a data copy and storing the original data and the data copy at the same time.

[0093] Since the first storage priority is for normal data, in order to ensure the accuracy of this data during subsequent transmission, a data copy is created in advance. If an error occurs during subsequent transmission, the data copy can be directly read out to ensure that subsequent operations can continue to run.

[0094] Step T2: Establish a second storage mechanism according to the second storage priority.

[0095] In the second storage mechanism, it includes: only storing the original data.

[0096] The storage unit 204 is used to initiate a storage request, confirm the data storage location according to the data storage mechanism, and complete the data storage.

[0097] Before confirming the data storage location according to the data storage mechanism, it also includes judging the storage request.

[0098] Specifically, after determining the storage mechanism, normal data and abnormal data can initiate a storage request. The storage request contains the data volume of the corresponding normal or abnormal data. Generally speaking, the storage request can be one or more, and each storage request can contain multiple types of data. For example, one storage request contains multiple normal data. Therefore, when there are multiple storage requests, the storage requests may be processed normally by the system or may not be processed normally. Therefore, specific judgments need to be made for multiple storage requests.

[0099] If there are K storage requests, the K storage requests are respectively defined as (1, 2,... k,... K). Assume that the data volumes contained in the m-th storage request are respectively j k1 ,j k2 ...j ki ,...j kI ,j ki indicating the data volume size of the i-th data contained in the k-th storage request.

[0100] Furthermore, different weights are assigned to different data volumes, where the weight represents the importance of this type of data.

[0101] The standard for assigning weights is: when the data volume of this data is greater than the specified threshold, a larger weight is assigned to it; when the data volume of this type of data is less than the specified threshold, a smaller weight is assigned to it. Among them, the weights assigned to the data volumes j k1 ,j k2 ...j ki ,...j kI are respectively ω k1 ,ω k2 ,...ω ki ,...ω KI , ω1 + ω2 +... ωp +... + ω I = 1, λ mn represents the weight value of the i-th data included in the k-th storage request.

[0102] The above weight assignment is pre-set by the staff, and the specific weight values are not limited here.

[0103] Among them, the processing standard U is determined according to the above parameters:

[0104]

[0105] Among them, q k represents the number of times the k-th storage request is successfully obtained, q' k represents the number of times the k-th storage request fails, and I represents the number of data included in the k-th storage request.

[0106] When U is greater than the specified threshold, it means that the system can no longer fully process the multiple storage requests obtained at this time, and the process exits. Otherwise, the data storage location is confirmed.

[0107] Among them, confirming the data storage location is to confirm the node for storing data, which specifically includes the following sub-steps:

[0108] Step R1: Determine the first storage node according to the storage request and the first storage priority, and store the normal data.

[0109] Among them, if the storage request is to request to store normal data, a specified number of first storage nodes are set.

[0110] Among them, in the first storage mechanism, since it is necessary to store data replicas and original data, in this embodiment, the standard for selecting the first storage node is: the selected first storage node is a node that does not currently store data, so that more storage space can be provided for storing data replicas and original data.

[0111] Step R2: Determine the second storage node according to the storage request and the second storage priority, and store the abnormal data.

[0112] Among them, since the second storage mechanism only needs to store the original data, in this embodiment, the standard for selecting the second storage node is: the selected second storage node is a node that has already stored some other data, as long as the memory of this node can accommodate the storage of the original data.

[0113] This application has the following beneficial effects:

[0114] Before storing data, this application can first make a judgment during data acquisition, making a judgment from the source to save execution time for subsequent operations. Moreover, during the subsequent storage process, different storage mechanisms are proposed according to different data, which improves the data storage time and the accuracy of data storage.

[0115] Although the examples referred to in the current application are described, they are for explanatory purposes only and not a limitation of the present application. Changes, additions, and / or deletions to the embodiments can be made without departing from the scope of the present application.

[0116] The above is only a specific embodiment of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art within the technical scope disclosed in the present application can easily think of changes or substitutions, which should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A storage optimization method, characterized in that, It includes the following steps: Obtain the data to be stored; Determine the storage priority of the obtained data; Establish a data storage mechanism according to the storage priority; Initiate a storage request, confirm the data storage location according to the data storage mechanism, and complete the data storage.

2. The storage optimization method according to claim 1, wherein Determining the storage priority of the obtained data includes the following sub-steps: Conduct data anomaly detection; Determine the storage priority according to the anomaly detection result.

3. The storage optimization method according to claim 2, characterized in that, Conducting data anomaly detection includes the following sub-steps: Judge whether there is an error in the obtained data; If there is an error in the obtained data, the process exits; If the obtained data is normal, judge whether there is an error in the data itself.

4. The storage optimization method according to claim 3, wherein If the data itself is normal, the anomaly detection result is normal; if the data itself is abnormal, the anomaly detection result is abnormal; Set the data with a normal detection result to the first storage priority, and set the data with an abnormal anomaly detection result to the second storage priority.

5. The storage optimization method according to claim 4, characterized in that, Establishing a data storage mechanism according to the storage priority includes the following sub-steps: Establish a first storage mechanism according to the first storage priority; Establish a second storage mechanism according to the second storage priority.

6. A storage optimization system, characterized in that, Specifically include: An acquisition unit, a priority determination unit, a storage mechanism establishment unit, and a storage unit; The acquisition unit is used to obtain the data to be stored; The priority determination unit is used to determine the storage priority of the obtained data; The storage mechanism establishment unit is used to establish a data storage mechanism according to the storage priority; The storage unit is used to initiate a storage request, confirm the data storage location according to the data storage mechanism, and complete the data storage.

7. The storage optimization system according to claim 6, characterized in that The priority determination unit, when determining the storage priority of the obtained data, includes the following sub-steps: Conduct data anomaly detection; Determine the storage priority according to the anomaly detection result.

8. The storage optimization system according to claim 7, wherein When the priority determination unit conducts data anomaly detection, it includes the following sub-steps: Judge whether there is an error in the obtained data; If there is an error in the obtained data, the process exits; If the obtained data is normal, judge whether there is an error in the data itself.

9. The storage optimization system according to claim 6, wherein In the priority determination unit, if the data itself is normal, the anomaly detection result is normal; if the data itself is abnormal, the anomaly detection result is abnormal; Set the data with a normal detection result to the first storage priority, and set the data with an abnormal anomaly detection result to the second storage priority.

10. The storage optimization system according to claim 9, wherein When the storage mechanism establishment unit establishes a data storage mechanism according to the storage priority, it includes the following sub-steps: Establish a first storage mechanism according to the first storage priority; Establish a second storage mechanism according to the second storage priority.

Images