A data storage method and system for customized solutions

By building data links and performing data correlation and break-chain optimization processing, the problems of massive data storage delay and low efficiency are solved, and efficient and secure data storage is achieved.

CN118567579BActive Publication Date: 2024-11-05SHANDONG STREET MOBILE SHOP NEW ENERGY TECH CO LTD
View PDF 9 Cites 0 Cited by

Patent Information

Application Number
CN202411037531.8
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-07-31
Publication Date
2024-11-05
Estimated Expiration
2044-07-31

AI Technical Summary

Technical Problem

When existing data storage methods face massive data stream storage requests, they lead to an increase in data storage latency, which cannot meet the growing data capacity and access speed requirements, resulting in insufficiency of storage.

Method used

A customized data storage method is proposed, by receiving data in a preset time period, building a data link, extracting the historical characteristics of header data and adjacent data, calculating factors, judging data associations, and performing a link break in the coordinated data storage chain Optimize processing to ensure efficient storage.

Benefits of technology

It effectively reduces data storage latency, improves storage efficiency, ensures data security and reliability, and adapts to massive data storage needs.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118567579B_ABST
    Figure CN118567579B_ABST
Patent Text Reader

Abstract

The present invention relates to the technical field of data storage, and discloses a data storage method and system for customized solutions. The method comprises the following steps: receiving data to be stored, constructing a data chain to be stored based on data receiving time, extracting head data to be stored and adjacent data to be stored on the data chain to be stored, calculating head data factor and adjacent data factor to be stored according to historical data characteristics, and judging whether to perform data association between the head data to be stored and the adjacent data to be stored; judging in turn whether the remaining data to be stored on the data chain to be stored is data associated with the head data to be stored, and determining a collaborative data storage chain; parsing a target storage device of the collaborative data storage chain, judging whether to perform chain-break optimization processing on the collaborative data storage chain based on the parsing result, and storing the collaborative data storage chain in the target storage device. The method can effectively cope with the storage of massive data, improve data storage efficiency, reduce data storage latency, and ensure data security and reliability.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of data storage, and in particular to a data storage method and system for customized solutions. Background Art

[0002] The current data storage method mainly stores the data to be stored in batches according to the storage request time. When a large number of data stream storage requests are received at the same time, the existing data storage method will greatly increase the data storage latency and cannot quickly store the data to be stored, which seriously reduces the data storage efficiency.

[0003] Therefore, the traditional single data storage method can no longer meet the growing needs of data capacity and access speed. Summary of the invention

[0004] In view of this, the present invention proposes a data storage method and system for customized solutions, aiming to solve the problem that the current technology cannot efficiently store massive data, cannot improve data storage efficiency, and causes high data storage delay.

[0005] The present invention proposes a data storage method for a customized solution, comprising:

[0006] Receive all data to be stored within a preset time period, and build a data chain to be stored based on the data receiving time, wherein every two data to be stored on the data chain to be stored are connected based on a data connection chain;

[0007] Extracting the head data to be stored and the adjacent data to be stored adjacent to the head data to be stored on the data chain to be stored, obtaining historical data features of the head data to be stored and the adjacent data to be stored, and calculating the corresponding head data factor to be stored and the adjacent data factor to be stored according to the historical data features;

[0008] Determining whether to perform data association between the head data to be stored and the adjacent data to be stored based on the head data to be stored factor and the adjacent data to be stored factor;

[0009] sequentially determining whether the remaining data to be stored on the data chain to be stored is associated with the head data to be stored, and determining a collaborative data storage chain based on the determination result;

[0010] The target storage device of the collaborative data storage chain is parsed, and based on the parsing result, it is determined whether to perform a disconnection optimization process on the collaborative data storage chain, and based on the determination result, the collaborative data storage chain is stored in the target storage device.

[0011] Furthermore, before receiving all the data to be stored within a preset time period, the method further includes:

[0012] Acquire original stored data, and perform data cleaning on the original stored data, wherein the data cleaning includes processing missing values, abnormal values, and duplicate values;

[0013] Performing data conversion on the cleaned original stored data, wherein the data conversion includes data standardization and data normalization;

[0014] Verify the integrity and consistency of the original stored data after data conversion, and determine whether it conforms to the expected format. If the verification passes and conforms to the expected format, the original stored data is sent as the data to be stored.

[0015] Further, when the historical data features of the head data to be stored and the adjacent data to be stored are obtained, and the corresponding head data factor to be stored and the adjacent data factor to be stored are calculated according to the historical data features, it includes:

[0016] Analyze the historical characteristic data of the head data to be stored, and determine the characteristic data value corresponding to each historical characteristic data;

[0017] Calculate the data allocation value corresponding to each feature data according to the feature data value;

[0018] ;

[0019] Wherein, a is the data allocation value corresponding to the characteristic data of the head data to be stored, e is a constant, b is the characteristic data value corresponding to the characteristic data, and Δb is the preset characteristic data value;

[0020] A first allocation interval, a second allocation interval and a third allocation interval are pre-set, wherein the data allocation range corresponding to the first allocation interval is [0, A1], the data allocation range corresponding to the second allocation interval is [A1, A2], and the data allocation range corresponding to the third allocation interval is [A2, A3];

[0021] Allocate all data allocation values ​​to corresponding allocation spaces, and count the first allocation quantity of the first allocation interval, the second allocation quantity of the second allocation interval, and the third allocation quantity of the third allocation interval;

[0022] Calculate a head data factor G1 of the head data to be stored based on the first allocated quantity, the second allocated quantity and the third allocated quantity;

[0023] Analyze the historical characteristic data of the adjacent data to be stored to determine the characteristic data value corresponding to each historical characteristic data;

[0024] Calculate the data allocation value corresponding to each feature data according to the feature data value;

[0025] Allocate all data allocation values ​​to corresponding allocation spaces, and count the fourth allocation quantity of the first allocation interval, the fifth allocation quantity of the second allocation interval, and the sixth allocation quantity of the third allocation interval;

[0026] The adjacent data to be stored factor G2 of the adjacent data to be stored is calculated based on the fourth allocated quantity, the fifth allocated quantity, and the sixth allocated quantity.

[0027] Further, when calculating the head data to be stored factor G1 of the head data to be stored based on the first allocated quantity, the second allocated quantity and the third allocated quantity, it includes:

[0028] The head data factor G1 of the head data to be stored is calculated according to the following formula:

[0029] ;

[0030] Wherein, F1 is the first allocated quantity, F2 is the second allocated quantity, F3 is the third allocated quantity, and ln is the sign of the logarithmic function.

[0031] Further, when determining whether to associate the head data to be stored with the adjacent data to be stored based on the head data to be stored factor and the adjacent data to be stored factor, it includes:

[0032] Determine the first data weight corresponding to the head data to be stored and the second data weight corresponding to the adjacent data to be stored respectively based on the data type-weight mapping table;

[0033] The data association factor between the first data to be stored and the adjacent data to be stored is calculated according to the following formula:

[0034] ;

[0035] Wherein, K is the data association factor between the first data to be stored and the adjacent data to be stored, y1 is the first data weight, y2 is the second data weight, and h is the adjustment coefficient;

[0036] Determining whether to perform data association between the head data to be stored and the adjacent data to be stored according to the relationship between the data association factor and the preset data association factor;

[0037] When the data association factor is less than the preset data association factor, it is determined not to perform data association between the head data to be stored and the adjacent data to be stored;

[0038] When the data association factor is greater than or equal to the preset data association factor, it is determined that data association is performed between the head data to be stored and the adjacent data to be stored.

[0039] Furthermore, the adjustment coefficient h is determined according to the following method:

[0040] Constructing a first characteristic data value set according to all characteristic data values ​​of the head data to be stored;

[0041] Constructing a second characteristic data value set according to all characteristic data values ​​of the adjacent data to be stored;

[0042] Calculating the covariance between the first feature data value set and the second feature data value set;

[0043] The difference coefficient between the first data to be stored and the adjacent data to be stored is calculated according to the following formula:

[0044] ;

[0045] Wherein, L is the coefficient of variation, q is the covariance, r1 is the variance corresponding to the first feature data value set, and r2 is the variance corresponding to the second feature data value set;

[0046] Presetting a first preset difference coefficient and a second preset difference coefficient;

[0047] Presetting a first preset adjustment coefficient, a second preset adjustment coefficient, and a third preset adjustment coefficient;

[0048] When the first preset condition is identified, the first preset adjustment coefficient is used as the adjustment coefficient h;

[0049] When the second preset condition is identified, the second preset adjustment coefficient is used as the adjustment coefficient h;

[0050] When the third preset condition is identified, the third preset adjustment coefficient is used as the adjustment coefficient h; wherein, the first preset condition is that the difference coefficient is less than the first preset difference coefficient, the second preset condition is that the difference coefficient is greater than or equal to the first preset difference coefficient and less than the second preset difference coefficient, and the third preset condition is that the difference coefficient is greater than or equal to the second preset difference coefficient.

[0051] Further, when judging whether to perform a disconnection optimization process on the collaborative data storage chain based on the analysis result, it includes:

[0052] Determine the storage area of ​​the target storage device, and determine the load rate of each storage area;

[0053] Calculating the carrying capacity of the target storage device according to the load rate;

[0054] Determining whether to perform a disconnection optimization process on the collaborative data storage chain based on the relationship between the carrying capacity and the preset carrying capacity;

[0055] When the carrying capacity is less than the preset carrying capacity, it is determined to perform a chain-breaking optimization process on the collaborative data storage chain, wherein when performing the chain-breaking optimization process, the number of chain-breaking of the collaborative data storage chain is determined based on a carrying capacity-chain-breaking number mapping table, and the chain breaking starts from the tail of the collaborative data storage chain;

[0056] When the carrying capacity is greater than or equal to the preset carrying capacity, it is determined not to perform the disconnection optimization process on the collaborative data storage chain.

[0057] Further, when calculating the carrying capacity of the target storage device according to the load rate, it includes:

[0058] The carrying capacity of the target storage device is calculated according to the following formula:

[0059] ;

[0060] Where S is the carrying capacity of the target storage device, n is the number of storage areas, and t i is the weight corresponding to the i-th storage area, U i is the load rate of the i-th storage area, and V is the standard load rate.

[0061] Compared with the prior art, the present invention has the following beneficial effects:

[0062] The present invention discloses a data storage method and system for customized solutions, which receive data to be stored, build a data chain to be stored based on data receiving time, extract head data to be stored and adjacent data to be stored on the data chain to be stored, calculate the head data factor to be stored and the adjacent data factor to be stored according to historical data characteristics, and judge whether to associate data between the head data to be stored and the adjacent data to be stored; judge in turn whether the remaining data to be stored on the data chain to be stored is associated with the head data to be stored, and determine a collaborative data storage chain; parse a target storage device of the collaborative data storage chain, and judge whether to perform chain-breaking optimization processing on the collaborative data storage chain based on the parsing result, and store the collaborative data storage chain in the target storage device, which can effectively cope with the storage of massive data, improve data storage efficiency, reduce data storage latency, and ensure data security and reliability.

[0063] On the other hand, the present application also provides a data storage system for a customized solution, comprising:

[0064] A construction module, used for receiving all data to be stored within a preset time period, and constructing a data chain to be stored based on the data receiving time, wherein every two data to be stored on the data chain to be stored are connected based on a data connection chain;

[0065] a calculation module, used to extract the head data to be stored and the adjacent data to be stored adjacent to the head data to be stored on the data chain to be stored, obtain the historical data features of the head data to be stored and the adjacent data to be stored, and calculate the corresponding head data factor to be stored and the adjacent data factor to be stored according to the historical data features;

[0066] A judgment module, used for judging whether to perform data association between the head data to be stored and the adjacent data to be stored based on the head data to be stored factor and the adjacent data to be stored factor;

[0067] A determination module, used to determine in turn whether the remaining data to be stored on the data chain to be stored is associated with the head data to be stored, and determine the collaborative data storage chain based on the determination result;

[0068] The storage module is used to parse the target storage device of the collaborative data storage chain, determine whether to perform chain-breaking optimization processing on the collaborative data storage chain based on the parsing result, and store the collaborative data storage chain to the target storage device based on the determination result.

[0069] Furthermore, it also includes:

[0070] A cleaning module, used to obtain original stored data and perform data cleaning on the original stored data, wherein the data cleaning includes processing missing values, abnormal values ​​and duplicate values;

[0071] A conversion module, used for performing data conversion on the cleaned original storage data, wherein the data conversion includes data standardization and data normalization;

[0072] The verification module is used to verify the integrity and consistency of the original stored data after data conversion, and determine whether it conforms to the expected format. If the verification passes and conforms to the expected format, the original stored data is sent as the data to be stored.

[0073] It is understandable that the data storage system and method for customized solutions provided above have the same beneficial effects and will not be described in detail here. BRIEF DESCRIPTION OF THE DRAWINGS

[0074] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present invention. Moreover, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:

[0075] Figure 1 A schematic diagram of a process flow of a data storage method for a customized solution provided in an embodiment of the present invention;

[0076] Figure 2 A schematic diagram of the structure of a data storage system for a customized solution provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0077] Exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments described herein. On the contrary, these embodiments are provided in order to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art. It should be noted that, in the absence of conflict, the embodiments of the present invention and the features described in the embodiments can be combined with each other. The present invention will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0078] like Figure 1 As shown, in some embodiments of the present application, this embodiment provides a data storage method for a customized solution, including:

[0079] S110: receiving all data to be stored within a preset time period, and constructing a data chain to be stored based on the data receiving time, wherein every two data to be stored on the data chain to be stored are connected based on a data connection chain;

[0080] In some embodiments of the present application, before receiving all the data to be stored within a preset time period, the method further includes:

[0081] Acquire original stored data, and perform data cleaning on the original stored data, wherein the data cleaning includes processing missing values, abnormal values, and duplicate values;

[0082] Performing data conversion on the cleaned original stored data, wherein the data conversion includes data standardization and data normalization;

[0083] Verify the integrity and consistency of the original stored data after data conversion, and determine whether it conforms to the expected format. If the verification passes and conforms to the expected format, the original stored data is sent as the data to be stored.

[0084] In this embodiment, the methods of data cleaning and data conversion are already complicated and mature, and will not be introduced in detail here.

[0085] In this embodiment, after data conversion, it is necessary to check whether the data conforms to the expected format and structure to ensure that no damaged or inconsistent data is introduced, including checking field length, data type, range limit, etc.

[0086] The beneficial effect of the above technical solution is that the present invention iterates and adjusts the original stored data multiple times to ensure that the quality of the data meets the requirements of the project.

[0087] S120: extracting the head data to be stored and the adjacent data to be stored adjacent to the head data to be stored on the data chain to be stored, obtaining historical data features of the head data to be stored and the adjacent data to be stored, and calculating the corresponding head data factor to be stored and the adjacent data factor to be stored according to the historical data features;

[0088] In some embodiments of the present application, when acquiring the historical data features of the head data to be stored and the adjacent data to be stored, and calculating the corresponding head data factor to be stored and the adjacent data factor to be stored according to the historical data features, it includes:

[0089] Analyze the historical characteristic data of the head data to be stored, and determine the characteristic data value corresponding to each historical characteristic data;

[0090] Calculate the data allocation value corresponding to each feature data according to the feature data value;

[0091] ;

[0092] Wherein, a is the data allocation value corresponding to the characteristic data of the head data to be stored, e is a constant, b is the characteristic data value corresponding to the characteristic data, and Δb is the preset characteristic data value;

[0093] A first allocation interval, a second allocation interval and a third allocation interval are pre-set, wherein the data allocation range corresponding to the first allocation interval is [0, A1], the data allocation range corresponding to the second allocation interval is [A1, A2], and the data allocation range corresponding to the third allocation interval is [A2, A3];

[0094] Allocate all data allocation values ​​to corresponding allocation spaces, and count the first allocation quantity of the first allocation interval, the second allocation quantity of the second allocation interval, and the third allocation quantity of the third allocation interval;

[0095] Calculate a head data factor G1 of the head data to be stored based on the first allocated quantity, the second allocated quantity and the third allocated quantity;

[0096] Analyze the historical characteristic data of the adjacent data to be stored to determine the characteristic data value corresponding to each historical characteristic data;

[0097] Calculate the data allocation value corresponding to each feature data according to the feature data value;

[0098] Allocate all data allocation values ​​to corresponding allocation spaces, and count the fourth allocation quantity of the first allocation interval, the fifth allocation quantity of the second allocation interval, and the sixth allocation quantity of the third allocation interval;

[0099] The adjacent data to be stored factor G2 of the adjacent data to be stored is calculated based on the fourth allocated quantity, the fifth allocated quantity, and the sixth allocated quantity.

[0100] In this embodiment, the historical feature data includes historical storage times, historical retrieval times, historical safe storage times, historical risk storage times, etc.

[0101] In this embodiment, A1=0.6, A2=0.9, A3=+∞.

[0102] The beneficial effects of the above technical solution are: the present invention calculates the head data factor G1 of the head data to be stored based on the first allocated quantity, the second allocated quantity and the third allocated quantity, and calculates the adjacent data factor G2 of the adjacent data to be stored based on the fourth allocated quantity, the fifth allocated quantity and the sixth allocated quantity, which can realize the accurate calculation of the head data factor and the adjacent data factor to be stored, avoid calculation errors, lay the foundation for data storage, and provide data support.

[0103] In some embodiments of the present application, when calculating the head data to be stored factor G1 of the head data to be stored based on the first allocated quantity, the second allocated quantity, and the third allocated quantity, it includes:

[0104] The head data factor G1 of the head data to be stored is calculated according to the following formula:

[0105] ;

[0106] Wherein, F1 is the first allocated quantity, F2 is the second allocated quantity, F3 is the third allocated quantity, and ln is the sign of the logarithmic function.

[0107] In this embodiment, the calculation method of the adjacent data to be stored factor is the same as the calculation method of the head data to be stored factor. The fourth allocated quantity, the fifth allocated quantity and the sixth allocated quantity are substituted into the calculation. In order to save space, they are not shown here.

[0108] S130: judging whether to associate the head data to be stored with the adjacent data to be stored based on the head data to be stored factor and the adjacent data to be stored factor;

[0109] In some embodiments of the present application, when determining whether to perform data association between the head data to be stored and the adjacent data to be stored based on the head data to be stored factor and the adjacent data to be stored factor, the method includes:

[0110] Determine the first data weight corresponding to the head data to be stored and the second data weight corresponding to the adjacent data to be stored respectively based on the data type-weight mapping table;

[0111] The data association factor between the first data to be stored and the adjacent data to be stored is calculated according to the following formula:

[0112] ;

[0113] Wherein, K is the data association factor between the first data to be stored and the adjacent data to be stored, y1 is the first data weight, y2 is the second data weight, and h is the adjustment coefficient;

[0114] Determining whether to perform data association between the head data to be stored and the adjacent data to be stored according to the relationship between the data association factor and the preset data association factor;

[0115] When the data association factor is less than the preset data association factor, it is determined not to perform data association between the head data to be stored and the adjacent data to be stored;

[0116] When the data association factor is greater than or equal to the preset data association factor, it is determined that data association is performed between the head data to be stored and the adjacent data to be stored.

[0117] In this embodiment, the data type-weight mapping table is set in advance, and each data type corresponds to a weight, and the AHP hierarchical analysis method, entropy value method, etc. can be used.

[0118] In this embodiment, the preset data association factor is obtained by analyzing the historical storage data and is a fixed value, preferably 6 here.

[0119] The beneficial effect of the above technical solution is: the present invention determines whether to associate the head data to be stored and the adjacent data to be stored according to the relationship between the data association factor and the preset data association factor, which can provide a judgment basis for whether the data can be stored collaboratively, improve data storage efficiency, and at the same time will not affect data stability. Two incompatible data to be stored will not be stored at the same time.

[0120] In some embodiments of the present application, the adjustment coefficient h is determined according to the following method:

[0121] Constructing a first characteristic data value set according to all characteristic data values ​​of the head data to be stored;

[0122] Constructing a second characteristic data value set according to all characteristic data values ​​of the adjacent data to be stored;

[0123] Calculating the covariance between the first feature data value set and the second feature data value set;

[0124] The difference coefficient between the first data to be stored and the adjacent data to be stored is calculated according to the following formula:

[0125] ;

[0126] Wherein, L is the coefficient of variation, q is the covariance, r1 is the variance corresponding to the first feature data value set, and r2 is the variance corresponding to the second feature data value set;

[0127] Presetting a first preset difference coefficient and a second preset difference coefficient;

[0128] Presetting a first preset adjustment coefficient, a second preset adjustment coefficient, and a third preset adjustment coefficient;

[0129] When the first preset condition is identified, the first preset adjustment coefficient is used as the adjustment coefficient h;

[0130] When the second preset condition is identified, the second preset adjustment coefficient is used as the adjustment coefficient h;

[0131] When the third preset condition is identified, the third preset adjustment coefficient is used as the adjustment coefficient h; wherein, the first preset condition is that the difference coefficient is less than the first preset difference coefficient, the second preset condition is that the difference coefficient is greater than or equal to the first preset difference coefficient and less than the second preset difference coefficient, and the third preset condition is that the difference coefficient is greater than or equal to the second preset difference coefficient.

[0132] In this embodiment, the covariance between 5, 6, 8, 9, 5 and 8, 9, 4, 2, 3 is -3.155, the variance of 5, 6, 8, 9, 5 is 2.6, and the variance of 8, 9, 4, 2, 3 is 8.04. The calculation process of covariance and variance is relatively mature, and the calculation process is not given as an example here.

[0133] In this embodiment, the first preset difference coefficient and the second preset difference coefficient are fixed values, which are preferably 0.45 and 0.65 respectively.

[0134] In this embodiment, the first preset adjustment coefficient, the second preset adjustment coefficient and the third preset adjustment coefficient are fixed values, which are preferably 1.25, 1 and 0.85 respectively.

[0135] The beneficial effect of the above technical solution is: the present invention sets the adjustment coefficient h according to the difference coefficient, the first preset difference coefficient and the second preset difference coefficient, which can realize dynamic adjustment of the data association factor, avoid the singleness of the calculation process, and further ensure the accuracy and comprehensiveness of the data association factor.

[0136] S140: sequentially determining whether the remaining data to be stored on the data chain to be stored are associated with the head data to be stored, and determining a collaborative data storage chain based on the determination result;

[0137] In this embodiment, the process of whether the remaining data to be stored is data-associated with the head data to be stored will not be repeated here. If there is data to be stored that is not data-associated with the head data to be stored, this data will be disconnected, and finally all the data to be stored that are data-associated with the head data to be stored will be data-linked.

[0138] S150: Analyze the target storage device of the collaborative data storage chain, determine whether to perform disconnection optimization processing on the collaborative data storage chain based on the analysis result, and store the collaborative data storage chain in the target storage device based on the determination result.

[0139] In some embodiments of the present application, when determining whether to perform a broken link optimization process on the collaborative data storage chain based on the analysis result, it includes:

[0140] Determine the storage area of ​​the target storage device, and determine the load rate of each storage area;

[0141] Calculating the carrying capacity of the target storage device according to the load rate;

[0142] Determining whether to perform a disconnection optimization process on the collaborative data storage chain based on the relationship between the carrying capacity and the preset carrying capacity;

[0143] When the carrying capacity is less than the preset carrying capacity, it is determined to perform a chain-breaking optimization process on the collaborative data storage chain, wherein when performing the chain-breaking optimization process, the number of chain-breaking of the collaborative data storage chain is determined based on a carrying capacity-chain-breaking number mapping table, and the chain breaking starts from the tail of the collaborative data storage chain;

[0144] When the carrying capacity is greater than or equal to the preset carrying capacity, it is determined not to perform the disconnection optimization process on the collaborative data storage chain.

[0145] In this embodiment, the load rate is an indicator reflecting the carrying capacity of the target storage device and is used to measure its operating status and efficiency. The load rate can be determined according to the ratio of the data transmission volume of the target storage device in a specific time to the total transmission capacity.

[0146] In this embodiment, the preset carrying capacity is set according to the historical storage situation of the target storage device, and is a value that can reflect the storage capacity of the target storage device, and is preferably 12 here.

[0147] In this embodiment, the carrying capacity-broken link number mapping table is set in advance, and the carrying capacity corresponds to the broken link number, which can ensure that the storage capacity of the target storage device is within a safe and stable range.

[0148] In this embodiment, the chain is broken starting from the end of the collaborative data storage chain.

[0149] The beneficial effect of the above technical solution is: the present invention determines whether to perform disconnection optimization processing on the collaborative data storage chain based on the relationship between the carrying capacity and the preset carrying capacity, which can not only ensure the effective storage of massive data in the same period, but also avoid the phenomenon of excessive storage pressure on the target storage device, resulting in data storage loss, thereby reducing data storage delay.

[0150] In some embodiments of the present application, when calculating the carrying capacity of the target storage device according to the load rate, it includes:

[0151] The carrying capacity of the target storage device is calculated according to the following formula:

[0152] ;

[0153] Where S is the carrying capacity of the target storage device, n is the number of storage areas, and t i is the weight corresponding to the i-th storage area, U i is the load rate of the i-th storage area, and V is the standard load rate.

[0154] like Figure 2 As shown, in another preferred embodiment based on the above embodiment, this embodiment provides a data storage system for a customized solution, including:

[0155] A construction module, used for receiving all data to be stored within a preset time period, and constructing a data chain to be stored based on the data receiving time, wherein every two data to be stored on the data chain to be stored are connected based on a data connection chain;

[0156] a calculation module, used to extract the head data to be stored and the adjacent data to be stored adjacent to the head data to be stored on the data chain to be stored, obtain the historical data features of the head data to be stored and the adjacent data to be stored, and calculate the corresponding head data factor to be stored and the adjacent data factor to be stored according to the historical data features;

[0157] A judgment module, used for judging whether to perform data association between the head data to be stored and the adjacent data to be stored based on the head data to be stored factor and the adjacent data to be stored factor;

[0158] A determination module, used to determine in turn whether the remaining data to be stored on the data chain to be stored is associated with the head data to be stored, and determine the collaborative data storage chain based on the determination result;

[0159] The storage module is used to parse the target storage device of the collaborative data storage chain, determine whether to perform chain-breaking optimization processing on the collaborative data storage chain based on the parsing result, and store the collaborative data storage chain to the target storage device based on the determination result.

[0160] In some embodiments of the present application, it also includes:

[0161] A cleaning module, used to obtain original stored data and perform data cleaning on the original stored data, wherein the data cleaning includes processing missing values, abnormal values ​​and duplicate values;

[0162] A conversion module, used for performing data conversion on the cleaned original storage data, wherein the data conversion includes data standardization and data normalization;

[0163] The verification module is used to verify the integrity and consistency of the original stored data after data conversion, and determine whether it conforms to the expected format. If the verification passes and conforms to the expected format, the original stored data is sent as the data to be stored.

[0164] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems or computer program products. Therefore, the present application may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0165] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems) and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0166] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0167] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0168] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the relevant field should understand that the specific implementation methods of the present invention can still be modified or replaced by equivalents. Any modification or equivalent replacement that does not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A data storage method for a customized solution, characterized in that: include: Receive all data to be stored within a preset time period, and build a data chain to be stored based on the data receiving time, wherein every two data to be stored on the data chain to be stored are connected based on a data connection chain; Extracting the head data to be stored and the adjacent data to be stored adjacent to the head data to be stored on the data chain to be stored, obtaining the historical data features of the head data to be stored and the adjacent data to be stored, and calculating the corresponding head data factor to be stored and the adjacent data factor to be stored according to the historical data features, wherein the historical feature data includes the number of historical storage times, the number of historical retrieval times, the number of historical safe storage times, and the number of historical risk storage times; Determining whether to perform data association between the head data to be stored and the adjacent data to be stored based on the head data to be stored factor and the adjacent data to be stored factor; sequentially determining whether the remaining data to be stored on the data chain to be stored is associated with the head data to be stored, and determining a collaborative data storage chain based on the determination result; Analyze the target storage device of the collaborative data storage chain, determine whether to perform a disconnection optimization process on the collaborative data storage chain based on the analysis result, and store the collaborative data storage chain in the target storage device based on the determination result; When the historical data features of the head data to be stored and the adjacent data to be stored are obtained, and the corresponding head data factor to be stored and the adjacent data factor to be stored are calculated according to the historical data features, it includes: Analyze the historical characteristic data of the head data to be stored, and determine the characteristic data value corresponding to each historical characteristic data; Calculate the data allocation value corresponding to each feature data according to the feature data value; ; Wherein, a is the data allocation value corresponding to the characteristic data of the head data to be stored, e is a constant, b is the characteristic data value corresponding to the characteristic data, and Δb is the preset characteristic data value; A first allocation interval, a second allocation interval and a third allocation interval are pre-set, wherein the data allocation range corresponding to the first allocation interval is [0, A1), the data allocation range corresponding to the second allocation interval is [A1, A2), and the data allocation range corresponding to the third allocation interval is [A2, A3], wherein A1=0.6, A2=0.9, A3=+∞; Allocate all data allocation values ​​to corresponding allocation spaces, and count the first allocation quantity of the first allocation interval, the second allocation quantity of the second allocation interval, and the third allocation quantity of the third allocation interval; Calculate a head data factor G1 of the head data to be stored based on the first allocated quantity, the second allocated quantity and the third allocated quantity; Analyze the historical characteristic data of the adjacent data to be stored to determine the characteristic data value corresponding to each historical characteristic data; Calculate the data allocation value corresponding to each feature data according to the feature data value; Allocate all data allocation values ​​to corresponding allocation spaces, and count the fourth allocation quantity of the first allocation interval, the fifth allocation quantity of the second allocation interval, and the sixth allocation quantity of the third allocation interval; Calculate the adjacent data to be stored factor G2 of the adjacent data to be stored based on the fourth allocated quantity, the fifth allocated quantity and the sixth allocated quantity; When calculating the head data to be stored factor G1 of the head data to be stored based on the first allocated quantity, the second allocated quantity and the third allocated quantity, it includes: The head data factor G1 of the head data to be stored is calculated according to the following formula: ; Wherein, F1 is the first allocated quantity, F2 is the second allocated quantity, F3 is the third allocated quantity, and ln is the sign of the logarithmic function; When judging whether to perform a disconnection optimization process on the collaborative data storage chain based on the analysis result, it includes: Determine the storage area of ​​the target storage device, and determine the load rate of each storage area; Calculating the carrying capacity of the target storage device according to the load rate; Determining whether to perform a disconnection optimization process on the collaborative data storage chain based on the relationship between the carrying capacity and the preset carrying capacity; When the carrying capacity is less than the preset carrying capacity, it is determined to perform a chain-breaking optimization process on the collaborative data storage chain, wherein when performing the chain-breaking optimization process, the number of chain-breaking of the collaborative data storage chain is determined based on a carrying capacity-chain-breaking number mapping table, and the chain breaking starts from the tail of the collaborative data storage chain; When the carrying capacity is greater than or equal to the preset carrying capacity, it is determined not to perform the disconnection optimization process on the collaborative data storage chain; When calculating the carrying capacity of the target storage device according to the load rate, it includes: The carrying capacity of the target storage device is calculated according to the following formula: ; Where S is the carrying capacity of the target storage device, n is the number of storage areas, and t i is the weight corresponding to the i-th storage area, U i is the load rate of the i-th storage area, and V is the standard load rate.

2. The data storage method for customized solutions according to claim 1, characterized in that: Before receiving all the data to be stored within a preset time period, it also includes: Acquire original stored data, and perform data cleaning on the original stored data, wherein the data cleaning includes processing missing values, abnormal values, and duplicate values; Performing data conversion on the cleaned original stored data, wherein the data conversion includes data standardization and data normalization; Verify the integrity and consistency of the original stored data after data conversion, and determine whether it conforms to the expected format. If the verification passes and conforms to the expected format, the original stored data is sent as the data to be stored.

3. The data storage method for customized solutions according to claim 1, characterized in that: When determining whether to associate data between the head data to be stored and the adjacent data to be stored based on the head data to be stored factor and the adjacent data to be stored factor, the method includes: Determine the first data weight corresponding to the head data to be stored and the second data weight corresponding to the adjacent data to be stored respectively based on the data type-weight mapping table; The data association factor between the first data to be stored and the adjacent data to be stored is calculated according to the following formula: ; Wherein, K is the data association factor between the first data to be stored and the adjacent data to be stored, y1 is the first data weight, y2 is the second data weight, and h is the adjustment coefficient; Determining whether to perform data association between the head data to be stored and the adjacent data to be stored according to the relationship between the data association factor and the preset data association factor; When the data association factor is less than the preset data association factor, it is determined not to perform data association between the head data to be stored and the adjacent data to be stored; When the data association factor is greater than or equal to the preset data association factor, it is determined that data association is performed between the head data to be stored and the adjacent data to be stored.

4. The data storage method for customized solutions according to claim 3, characterized in that: The adjustment factor h is determined according to the following method: Constructing a first characteristic data value set according to all characteristic data values ​​of the head data to be stored; Constructing a second characteristic data value set according to all characteristic data values ​​of the adjacent data to be stored; Calculating the covariance between the first feature data value set and the second feature data value set; The difference coefficient between the first data to be stored and the adjacent data to be stored is calculated according to the following formula: ; Wherein, L is the coefficient of variation, q is the covariance, r1 is the variance corresponding to the first feature data value set, and r2 is the variance corresponding to the second feature data value set; Presetting a first preset difference coefficient and a second preset difference coefficient; Presetting a first preset adjustment coefficient, a second preset adjustment coefficient, and a third preset adjustment coefficient; When the first preset condition is identified, the first preset adjustment coefficient is used as the adjustment coefficient h; When the second preset condition is identified, the second preset adjustment coefficient is used as the adjustment coefficient h; When the third preset condition is identified, the third preset adjustment coefficient is used as the adjustment coefficient h; wherein, the first preset condition is that the difference coefficient is less than the first preset difference coefficient, the second preset condition is that the difference coefficient is greater than or equal to the first preset difference coefficient and less than the second preset difference coefficient, and the third preset condition is that the difference coefficient is greater than or equal to the second preset difference coefficient.

5. A data storage system for customized solutions, applied to the data storage method for customized solutions according to any one of claims 1 to 4, characterized in that: include: A construction module, used for receiving all data to be stored within a preset time period, and constructing a data chain to be stored based on the data receiving time, wherein every two data to be stored on the data chain to be stored are connected based on a data connection chain; A calculation module, used to extract the head data to be stored and the adjacent data to be stored adjacent to the head data to be stored on the data chain to be stored, obtain the historical data features of the head data to be stored and the adjacent data to be stored, and calculate the corresponding head data factor to be stored and the adjacent data factor to be stored according to the historical data features, wherein the historical feature data includes the number of historical storage times, the number of historical retrieval times, the number of historical safe storage times, and the number of historical risk storage times; A judgment module, used for judging whether to perform data association between the head data to be stored and the adjacent data to be stored based on the head data to be stored factor and the adjacent data to be stored factor; A determination module, used to determine in turn whether the remaining data to be stored on the data chain to be stored is associated with the head data to be stored, and determine the collaborative data storage chain based on the determination result; The storage module is used to parse the target storage device of the collaborative data storage chain, determine whether to perform chain-breaking optimization processing on the collaborative data storage chain based on the parsing result, and store the collaborative data storage chain to the target storage device based on the determination result.

6. The data storage system for customized solutions according to claim 5, further comprising: A cleaning module, used to obtain original stored data and perform data cleaning on the original stored data, wherein the data cleaning includes processing missing values, abnormal values ​​and duplicate values; A conversion module, used for performing data conversion on the cleaned original storage data, wherein the data conversion includes data standardization and data normalization; The verification module is used to verify the integrity and consistency of the original stored data after data conversion, and determine whether it conforms to the expected format. If the verification passes and conforms to the expected format, the original stored data is sent as the data to be stored.

Citation Information

Patent Citations

  • Method and device for determining broken link data and nonvolatile storage medium

    CN112749167A

  • Method, device and equipment for distributing data links in distributed storage system

    CN114064288A

  • Data storage method, device and medium based on streaming processing

    CN114064647A

  • Mass offline data trusted storage method and system based on block chain

    CN114741735A

  • Information vulnerability processing method and device based on block chain

    CN116226138A