Big data storage method and system

By setting the security level and access frequency of data blocks in the HDFS storage system, using whale optimization algorithm to determine the optimal placement strategy, and dynamically adjust the number of copies, the high retrieval time caused by improper placement of data blocks in the distributed data storage system is solved. and the problem of insufficient security achieves efficient and secure data storage.

CN118535652BActive Publication Date: 2024-11-01ZHUO SHI ZHI XING (QINGTIAN) YUAN UNIVERSE TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411002286.7
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-07-25
Publication Date
2024-11-01
Estimated Expiration
2044-07-25

AI Technical Summary

Technical Problem

The existing distributed data storage system uses load balancing as the standard when placing data blocks, resulting in data blocks being placed on storage nodes that may be placed at far distances, increasing data reading time, and lacking flexible security policies, which can easily lead to sensitive data. Give way.

Method used

Using the HDFS storage system based on MaaS mode, by setting the security level and access frequency of data blocks, using whale optimization algorithm to determine the optimal data block placement strategy, and dynamically adjust the number of replicas of data blocks in the storage node to meet security needs. and minimize search time.

Benefits of technology

The data block placement strategy is optimized, data retrieval efficiency and security are improved, and the number of replicas is dynamically adjusted to improve the utilization efficiency of storage resources and reduce maintenance costs.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118535652B_ABST
    Figure CN118535652B_ABST
Patent Text Reader

Abstract

The present invention provides a big data storage method and system, and relates to the field of data processing technology. The method includes: constructing an HDFS storage system based on the MaaS mode; obtaining data to be stored; setting the security level of the data to be stored; processing the data to be stored in blocks to obtain multiple data blocks; determining the best data block placement strategy with the security level as a constraint and the retrieval time as a goal; storing each data block in a distributed manner on a storage node of the HDFS storage system according to the best data block placement strategy; analyzing the access frequency of each data block; and dynamically adjusting the number of copies of each data block in the storage node according to the access frequency of each data block. The present invention optimizes the data block placement strategy to ensure that the data retrieval time is minimized while meeting the security requirements, thereby improving the access efficiency of the system.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a big data storage method and system. Background Art

[0002] With the rapid development of big data technology, data has shown an exponential growth, and distributed storage systems have developed rapidly.

[0003] Current distributed data storage systems often place data blocks on storage nodes with lower loads based on data flow and load balancing as the placement criteria.

[0004] However, when load balancing is used as the placement strategy, data blocks may be placed in storage nodes that are far away from each other, which will lead to long search time and low data reading efficiency when reading data. At the same time, the current distributed data storage system lacks flexible security strategies and cannot flexibly adjust the security level according to the sensitivity of the data, which easily leads to sensitive data leakage. Summary of the invention

[0005] In order to solve the technical problems in the prior art that data blocks may be placed in storage nodes that are far away from each other, which will lead to long search time when reading data, low data reading efficiency, lack of flexible security strategies, and inability to flexibly adjust the security level according to the sensitivity of the data, which may easily lead to sensitive data leakage, the present invention provides a big data storage method and system.

[0006] The technical solution provided by the embodiment of the present invention is as follows:

[0007] First aspect

[0008] An embodiment of the present invention provides a big data storage method, including:

[0009] S1: Build an HDFS storage system based on the MaaS model;

[0010] S2: Get the data to be stored;

[0011] S3: Setting the security level of the data to be stored;

[0012] S4: dividing the data to be stored into blocks to obtain multiple data blocks;

[0013] S5: Determine the best data block placement strategy with the security level as the constraint and the retrieval time as the goal;

[0014] S6: According to the optimal data block placement strategy, each data block is distributedly stored on the storage nodes of the HDFS storage system;

[0015] S7: Analyze the access frequency of each data block;

[0016] S8: Dynamically adjust the number of copies of each data block in the storage node according to the access frequency of each data block.

[0017] Second aspect

[0018] An embodiment of the present invention provides a big data storage system, including:

[0019] processor;

[0020] A memory having computer-readable instructions stored thereon, wherein when the computer-readable instructions are executed by the processor, the big data storage method as described in the first aspect is implemented.

[0021] The third aspect

[0022] An embodiment of the present invention provides a computer-readable storage medium, on which a computer program is stored. When the program is executed by a processor, the big data storage method as described in the first aspect is implemented.

[0023] The beneficial effects brought about by the technical solution provided by the embodiment of the present invention include at least:

[0024] (1) In the present invention, the optimal data block placement strategy is determined with the security level as a constraint and the retrieval time as a goal. According to the optimal data block placement strategy, each data block is distributedly stored on the storage nodes of the HDFS storage system. By optimizing the data block placement strategy, it is ensured that the data retrieval time is minimized while meeting the security requirements, thereby improving the access efficiency of the system.

[0025] (2) In the present invention, the security level of the data to be stored can be flexibly set, and then the optimal data block placement strategy is determined based on the security level as a constraint, providing a flexible security strategy and improving data security.

[0026] (3) In the present invention, the number of copies of each data block in the storage node is dynamically adjusted according to the access frequency of each data block, and copies are added to the frequently accessed data blocks to improve their availability and access performance. The number of copies is reduced for the less frequently accessed data blocks, saving storage space and maintenance costs, thereby ensuring efficient use of storage resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0028] Figure 1 A flowchart of a big data storage method provided by an embodiment of the present invention;

[0029] Figure 2 A schematic diagram of the structure of a data block placement strategy provided by an embodiment of the present invention;

[0030] Figure 3 A schematic diagram of a flow chart of a method for dynamically adjusting the number of replicas provided in an embodiment of the present invention;

[0031] Figure 4 A schematic diagram of the structure of a big data storage system provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0032] The technical solution of the present invention is described below in conjunction with the accompanying drawings.

[0033] In the embodiments of the present invention, words such as "exemplarily" and "for example" are used to provide examples, illustrations or explanations. Any embodiment or design described as "example" in the present invention should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of the word "example" is intended to present the concept in a specific way. In addition, in the embodiments of the present invention, the meaning expressed by "and / or" can be both, or it can be either of the two.

[0034] In the embodiments of the present invention, "image" and "picture" can sometimes be used interchangeably. It should be noted that when the difference between them is not emphasized, the meanings they intend to express are the same. "of", "corresponding, relevant" and "corresponding" can sometimes be used interchangeably. It should be noted that when the difference between them is not emphasized, the meanings they intend to express are the same.

[0035] In the embodiments of the present invention, sometimes a subscript such as W1 may be written as a non-subscript such as W1. When the difference is not emphasized, the meanings to be expressed are the same.

[0036] In order to make the technical problems, technical solutions and advantages to be solved by the present invention more clear, a detailed description will be given below with reference to the accompanying drawings and specific embodiments.

[0037] Reference Manual Attached Figure 1 , showing a flow chart of a big data storage method provided by an embodiment of the present invention.

[0038] The embodiment of the present invention provides a big data storage method, which can be implemented by a big data storage device, which can be a terminal or a server. The processing flow of the big data storage method may include the following steps:

[0039] S1: Build an HDFS storage system based on the MaaS model.

[0040] Among them, MaaS (Model as a Service) is a cloud computing service model that provides various data models as services to users. Users can call these models through APIs or other interfaces to perform tasks such as data processing, analysis, and prediction. The MaaS model abstracts complex model management, maintenance, and execution into services, allowing users to focus on business applications without having to worry about the implementation and deployment details of the underlying models.

[0041] Among them, the HDFS (Hadoop Distributed File System) storage system is part of the Apache Hadoop project and is a distributed file system designed for large-scale data storage and processing. HDFS can build a highly scalable and reliable storage system with low-cost hardware, specifically for processing large data sets.

[0042] Combining the MaaS model with the HDFS storage system, an efficient big data storage and processing platform can be built. The specific implementation steps include: deploying HDFS on a distributed cluster, configuring NameNode, DataNode, and SecondaryNameNode to ensure high availability and reliability of the system. Deploy the MaaS platform on top of HDFS and provide various data processing and analysis models to users as services. Utilize the block storage and replication mechanism of HDFS to efficiently store large data sets on distributed nodes to ensure data reliability and scalability. Through the API interface, various data models are made into services, and users can easily call these services for data processing and analysis. Utilize the elastic expansion capabilities of cloud computing to dynamically adjust storage and computing resources according to actual needs to ensure efficient operation of the system.

[0043] S2: Get the data to be stored.

[0044] S3: Set the security level of the data to be stored.

[0045] Among them, users can flexibly set the security level of the data to be stored according to actual needs.

[0046] Optionally, the security level is used to limit the minimum distance between storage nodes storing two data blocks of the same data. The higher the security level, the greater the minimum distance between storage nodes storing two data blocks of the same data.

[0047] In the present invention, the security level of the data to be stored can be flexibly set, and then the best data block placement strategy is determined based on the security level as a constraint, providing a flexible security strategy and improving data security.

[0048] S4: Divide the data to be stored into blocks to obtain multiple data blocks.

[0049] Specifically, the size of each data block is predetermined (e.g., 64MB or 128MB). This size can be adjusted according to the system configuration and specific usage scenarios. The read data is then divided into blocks according to the predetermined block size. Each block is processed independently and marked as an independent data block.

[0050] S5: Determine the best data block placement strategy with the security level as the constraint and the retrieval time as the goal.

[0051] In the present invention, the optimal data block placement strategy is determined with the security level as a constraint and the retrieval time as a goal. According to the optimal data block placement strategy, each data block is distributedly stored on the storage node of the HDFS storage system. By optimizing the data block placement strategy, it is ensured that the data retrieval time is minimized while meeting the security requirements, thereby improving the access efficiency of the system.

[0052] Optionally, the data block placement strategy specifically includes: a storage node selection sub-strategy and a corresponding data block placement sub-strategy.

[0053] The storage node selection sub-strategy is as follows: ,s i Indicates whether the i-th storage node is selected to store data blocks. When , it means that the i-th storage node is selected to store the data block. , it means that the i-th storage node is not selected to store the data block, and n represents the total number of storage nodes.

[0054] The data block placement sub-strategy is as follows: , c i Represents the size of the data block stored in the i-th storage node. hour, ,and

[0055]

[0056] Wherein, C represents the size of the data to be stored.

[0057] Reference Manual Attached Figure 2 , showing a structural schematic diagram of a data block placement strategy provided by an embodiment of the present invention.

[0058] Figure 2 In the figure, the rectangle represents a complete data to be stored, which is divided into multiple data blocks (small squares). The circle represents a storage node, and multiple data blocks are placed in multiple distributed storage nodes.

[0059] Optionally, the storage node to which the user submits a data read request is access point p, and other storage nodes storing data to be read need to transmit the stored data blocks to access point p using the shortest path transmission method. The specific retrieval time is:

[0060]

[0061] Among them, Tr represents the retrieval time, c i represents the size of the data block stored in the i-th storage node, n represents the total number of storage nodes, L ip represents the shortest path between the ith storage node and access point p, and the shortest path L between the ith storage node and access point p ip Including multiple transmission links l, B l Represents the bandwidth on the lth transmission link.

[0062] Specifically, the classic Dijkstra algorithm may be used to determine the shortest path between the storage node and the access point. This part is already a prior art, and the present invention will not elaborate on it in order to avoid repetition.

[0063] In a possible implementation, S5 specifically includes: determining the best data block placement strategy through the whale optimization algorithm with the security level being constrained and the retrieval time being minimized.

[0064] Among them, the Whale Optimization Algorithm (WOA) is a swarm intelligence optimization algorithm based on whale hunting behavior, which simulates the hunting strategy of humpback whales, especially their "bubble net" hunting technology. WOA is mainly used to solve various complex optimization problems and has a wide range of applications in the fields of continuous optimization and discrete optimization.

[0065] Specifically, the inverse of the retrieval time is used as the fitness function of the whale optimization algorithm.

[0066] Initialize the whale population, which contains multiple whale individuals. The position of each whale individual in the search space represents a data block placement strategy.

[0067] Generate a random number r1 between 0 and 1 and calculate the first coefficient A:

[0068]

[0069] Where a represents the adaptive convergence factor.

[0070] Optionally, the adaptive convergence factor a is specifically:

[0071]

[0072] Among them, t represents the current iteration number, and T represents the maximum iteration number.

[0073] It should be noted that in the first half of the iteration, the adaptive convergence factor a gradually decreases from 2 to 0. This allows the individual whales to explore a wider search space in the early stage, avoid falling into the local optimal solution, and enhance the global search ability. In the second half of the iteration, the adaptive convergence factor a gradually decreases from 0 to -1, so that the individual whales gradually converge to the vicinity of the global optimal solution during the search process, enhance the local search ability, and improve the convergence speed of the algorithm.

[0074] when Go to the next step. , enter the random search mechanism.

[0075] Generates a random variable p.

[0076] when When , the shrinking and encircling mechanism is entered. When the spiral mechanism is entered.

[0077] Determine the current global optimal position, and other whales move towards the global optimal position, and update their own positions through the shrinking and surrounding mechanism:

[0078]

[0079] Among them, X(t+1) represents the individual position of the whale at the t+1th iteration, X best (t) represents the global optimal position at the tth iteration, A represents the first coefficient, D represents the randomized distance between the individual position of the whale and the current global optimal position, C represents the second coefficient, t represents the current iteration number, and X(t) represents the individual position of the whale at the tth iteration.

[0080] It should be noted that by updating the individual positions of whales through the shrinking and surrounding mechanism, the whale optimization algorithm can achieve a balance between global search and local search during the search process, enhance the exploration and development capabilities of the algorithm, improve the convergence speed and solution accuracy, and thus find the global optimal solution more efficiently.

[0081] Generate a random number r2 between 0 and 1. The whale moves in an upward spiral and continuously shrinks the encirclement, updating its position through the spiral mechanism:

[0082]

[0083] in, represents the distance between the individual position of the whale and the current global optimal position, e represents a natural constant, b represents a logarithmic spiral constant, and l represents a random number between -1 and 1.

[0084] It should be noted that different spiral paths (cosine or sine) are selected according to the value of the random number r2 to increase the diversity of the paths. By moving in an upward spiral and continuously shrinking the encirclement, the whale optimization algorithm can effectively balance global search and local optimization, improve the diversity and flexibility of the search path, enhance the robustness and adaptability of the algorithm, quickly approach the global optimal solution, and ultimately improve the optimization effect and efficiency. This method combines the spiral mechanism and the shrinking encirclement mechanism, and further enhances the comprehensiveness and effectiveness of the search by randomly selecting the movement mode.

[0085] Based on the positions of individual whales relative to each other, they update their own positions through a random search mechanism:

[0086]

[0087] Among them, X rand Represents the position of a randomly selected individual whale.

[0088] It should be noted that by updating the position of individual whales through a random search mechanism, the whale optimization algorithm can enhance the global search capability and exploration diversity during the search process, avoid premature convergence, and improve the robustness and adaptability of the algorithm. When applied to the optimization of data block placement strategies, it can significantly improve the performance, reliability, and resource utilization efficiency of distributed storage systems.

[0089] Update the global optimal position.

[0090] Generate a random number r3 and compare it with the mutation probability. If the random number is smaller than the mutation probability, the current position of the individual whale is mutated:

[0091]

[0092] Among them, X worst (t) represents the global worst position at the tth iteration.

[0093] In the present invention, by introducing a mutation mechanism and performing mutation processing on the positions of individual whales according to the comparison results of random numbers and mutation probabilities, the diversity and global search capabilities of the whale optimization algorithm can be significantly enhanced.

[0094] Determine whether the current number of iterations has reached the maximum number of iterations. If so, output the data block placement strategy represented by the whale individual with the highest current fitness as the optimal data block placement strategy. Otherwise, return to continue iterating.

[0095] S6: According to the optimal data block placement strategy, each data block is distributed and stored on the storage nodes of the HDFS storage system.

[0096] S7: Analyze the access frequency of each data block.

[0097] Optionally, the access frequency of the data block is specifically:

[0098]

[0099] Among them, F t represents the access frequency of the data block at the end of the current tth access cycle, f t-1 represents the access frequency of the data block at the end of the t-1th access cycle, N t It represents the number of accesses to the data block in the current t-th access cycle, T represents the duration of the unit access cycle, and α represents the balance factor.

[0100] Among them, those skilled in the art can set the size according to actual conditions, and the present invention does not limit it.

[0101] In the present invention, the weighted moving average method combines historical data and current data to make the frequency calculation more comprehensive. Historical data provides long-term trends, while current data reflects the immediate situation. The combination of the two can more accurately predict future access frequencies, effectively reduce the fluctuation of access frequencies, and make frequency changes smoother.

[0102] S8: Dynamically adjust the number of copies of each data block in the storage node according to the access frequency of each data block.

[0103] In the present invention, the number of copies of each data block in the storage node is dynamically adjusted according to the access frequency of each data block, and copies are added to the frequently accessed data blocks to improve their availability and access performance. The copies are reduced for the less frequently accessed data blocks to save storage space and maintenance costs, thereby ensuring the efficient use of storage resources.

[0104] Reference Manual Attached Figure 3 , showing a flow chart of a method for dynamically adjusting the number of replicas provided in an embodiment of the present invention.

[0105] In a possible implementation, S8 specifically includes:

[0106] S801: Obtain the current number of copies of each data block.

[0107] S802: Calculate the access frequency of the current copies of each data block.

[0108] Optionally, the access frequency of the current copies of the data block is specifically:

[0109]

[0110] Among them, F ct represents the access frequency of existing copies at the end of the current t-th access cycle, F t It indicates the access frequency of the data block at the end of the current t-th access cycle, and m indicates the number of existing copies.

[0111] S803: Between the minimum number of copies and the maximum number of copies, dynamically adjust the number of copies of each data block in the storage node according to the access frequency of the current existing copies of each data block.

[0112] In a possible implementation manner, S803 specifically includes:

[0113] The 25% of data blocks with the highest access frequency among the existing copies are regarded as hot data blocks, and the number of copies of the hot data blocks is increased.

[0114] Among them, those skilled in the art can set the number of copies added each time according to actual conditions, for example, adding 1 copy each time, and the present invention is not limited thereto.

[0115] The 25% of data blocks with the lowest access frequency of the current existing copies are used as non-hotspot data blocks to reduce the number of copies of non-hotspot data blocks.

[0116] Among them, those skilled in the art can set the number of copies to be reduced each time according to actual conditions, for example, reducing 1 copy each time, and the present invention is not limited thereto.

[0117] In the present invention, hot data blocks are frequently accessed data blocks. By increasing the number of their copies, access requests can be dispersed to more storage nodes, reducing the load of each node, thereby speeding up data access and improving the response time of the system. Non-hot data blocks are less frequently accessed data blocks. Reducing the number of their copies can save storage space and network bandwidth, and these resources can be allocated to hot data blocks that need more copies, thereby improving the overall resource utilization of the system.

[0118] Optionally, the maximum number of copies is determined as follows:

[0119] Ensure that the replica update bandwidth is smaller than the system bandwidth:

[0120]

[0121] Among them, B update represents the replica update bandwidth, m represents the number of replicas, c u Indicates the size of a single copy, f updateIndicates the replica update frequency per unit time.

[0122] Solve the above inequality to determine the maximum number of replicas.

[0123] In the present invention, ensuring that the replica update bandwidth is smaller than the system bandwidth can avoid system overload caused by replica update, thereby ensuring that the system can still run stably under high load conditions.

[0124] Optionally, the minimum number of copies is determined as follows:

[0125] Get the user's expected availability probability P a .

[0126] When all copies of a data block are unavailable, the data block becomes unavailable. The unavailability probability of each data block is calculated:

[0127]

[0128] Among them, P i represents the unavailability probability of the i-th data block, P ij represents the unavailability probability of the jth copy of the ith data block, and m represents the number of copies.

[0129] When one data block of the target data is unavailable, the entire target data becomes unavailable. The unavailability probability of each target data is calculated:

[0130]

[0131] Among them, P f represents the probability of unavailability of the target data, and n represents the total number of data blocks.

[0132] Ensure availability probability of meeting user expectations:

[0133]

[0134] in, represents the number of combinations of selecting i elements from n elements, P a Represents the availability probability expected by the user.

[0135] Solve the above inequality to determine the minimum number of replicas.

[0136] In the present invention, it is possible to ensure that the minimum number of copies is determined under the premise of satisfying the user's expected availability probability, thereby optimizing system resource utilization and providing efficient and reliable data storage services.

[0137] The beneficial effects brought about by the technical solution provided by the embodiment of the present invention include at least:

[0138] (1) In the present invention, the optimal data block placement strategy is determined with the security level as a constraint and the retrieval time as a goal. According to the optimal data block placement strategy, each data block is distributedly stored on the storage nodes of the HDFS storage system. By optimizing the data block placement strategy, it is ensured that the data retrieval time is minimized while meeting the security requirements, thereby improving the access efficiency of the system.

[0139] (2) In the present invention, the security level of the data to be stored can be flexibly set, and then the optimal data block placement strategy is determined based on the security level as a constraint, providing a flexible security strategy and improving data security.

[0140] (3) In the present invention, the number of copies of each data block in the storage node is dynamically adjusted according to the access frequency of each data block, and copies are added to the frequently accessed data blocks to improve their availability and access performance. The number of copies is reduced for the less frequently accessed data blocks, saving storage space and maintenance costs, thereby ensuring efficient use of storage resources.

[0141] Reference Manual Attached Figure 4 , showing a structural schematic diagram of a big data storage system provided by an embodiment of the present invention.

[0142] The present invention further provides a big data storage system 20, which is applied to the above-mentioned big data storage method, comprising:

[0143] Processor 201.

[0144] The memory 202 stores computer-readable instructions. When the computer-readable instructions are executed by the processor 201, the big data storage method of the method embodiment is implemented.

[0145] The big data storage system 20 provided by the present invention can execute the above-mentioned big data storage method and achieve the same or similar technical effects. To avoid repetition, the present invention will not go into details.

[0146] The beneficial effects brought about by the technical solution provided by the embodiment of the present invention include at least:

[0147] (1) In the present invention, the optimal data block placement strategy is determined with the security level as a constraint and the retrieval time as a goal. According to the optimal data block placement strategy, each data block is distributedly stored on the storage nodes of the HDFS storage system. By optimizing the data block placement strategy, it is ensured that the data retrieval time is minimized while meeting the security requirements, thereby improving the access efficiency of the system.

[0148] (2) In the present invention, the security level of the data to be stored can be flexibly set, and then the optimal data block placement strategy is determined based on the security level as a constraint, providing a flexible security strategy and improving data security.

[0149] (3) In the present invention, the number of copies of each data block in the storage node is dynamically adjusted according to the access frequency of each data block, and copies are added to the frequently accessed data blocks to improve their availability and access performance. The number of copies is reduced for the less frequently accessed data blocks, saving storage space and maintenance costs, thereby ensuring efficient use of storage resources.

[0150] It should be understood that the processor in the embodiment of the present invention may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0151] It should also be understood that the memory in the embodiments of the present invention may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic random access memory (DRAM), synchronous DRAM (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0152] The above embodiments can be implemented in whole or in part by software, hardware (such as circuits), firmware or any other combination. When implemented by software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center by wired (such as infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that contains one or more available media sets. The available medium can be a magnetic medium (for example, a floppy disk, a hard disk, a tape), an optical medium (for example, a DVD), or a semiconductor medium. The semiconductor medium can be a solid-state hard disk.

[0153] It should be understood that the term "and / or" in this article is only a description of the association relationship of the associated objects, which can exist in three relationships. For example, A and / or B can be: A exists alone, A and B exist at the same time, and B exists alone. A and B can be singular or plural. In addition, the character " / " in this article generally means that the associated objects before and after are in an "or" relationship, but it may also be an "and / or" relationship. Please refer to the context for specific understanding.

[0154] In the present invention, "at least one" means one or more, and "more" means two or more. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can be: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple.

[0155] It should be understood that in various embodiments of the present invention, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0156] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0157] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described equipment, devices and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0158] In the several embodiments provided by the present invention, it should be understood that the disclosed devices, apparatuses and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0159] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0160] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0161] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, and other media that can store program codes.

[0162] An embodiment of the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein the program, when executed by a processor, implements the big data storage method as described in the method embodiment.

[0163] A computer-readable storage medium provided by the present invention can implement the steps and effects of the big data storage method in the above method embodiment. To avoid repetition, the present invention will not go into details.

[0164] The beneficial effects brought about by the technical solution provided by the embodiment of the present invention include at least:

[0165] (1) In the present invention, the optimal data block placement strategy is determined with the security level as a constraint and the retrieval time as a goal. According to the optimal data block placement strategy, each data block is distributedly stored on the storage nodes of the HDFS storage system. By optimizing the data block placement strategy, it is ensured that the data retrieval time is minimized while meeting the security requirements, thereby improving the access efficiency of the system.

[0166] (2) In the present invention, the security level of the data to be stored can be flexibly set, and then the optimal data block placement strategy is determined based on the security level as a constraint, providing a flexible security strategy and improving data security.

[0167] (3) In the present invention, the number of copies of each data block in the storage node is dynamically adjusted according to the access frequency of each data block, and copies are added to the frequently accessed data blocks to improve their availability and access performance. The number of copies is reduced for the less frequently accessed data blocks, saving storage space and maintenance costs, thereby ensuring efficient use of storage resources.

[0168] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

[0169] There are a few points to note:

[0170] (1) The drawings of the embodiments of the present invention only involve structures related to the embodiments of the present invention. Other structures may refer to conventional designs.

[0171] (2) For the sake of clarity, in the drawings used to describe the embodiments of the present invention, the thickness of layers or regions is exaggerated or reduced, that is, these drawings are not drawn according to the actual scale. It is understood that when an element such as a layer, film, region or substrate is referred to as being "on" or "under" another element, the element may be "directly" "on" or "under" the other element or there may be intermediate elements.

[0172] (3) In the absence of conflict, the embodiments of the present invention and the features therein may be combined with each other to obtain new embodiments.

[0173] The above are only specific embodiments of the present invention, but the protection scope of the present invention is not limited thereto. The protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A big data storage method, characterized in that: include: S1: Build an HDFS storage system based on the MaaS model; S2: Get the data to be stored; S3: Setting the security level of the data to be stored; S4: dividing the data to be stored into blocks to obtain multiple data blocks; S5: Determine the best data block placement strategy with the security level as the constraint and the retrieval time as the goal; S6: According to the optimal data block placement strategy, each data block is distributedly stored on the storage nodes of the HDFS storage system; S7: Analyze the access frequency of each data block; The access frequency of the data block is specifically: ; in, F t Indicates the current t The access frequency of the data block at the end of the access cycle, f t-1 Indicates t - the access frequency of the data block at the end of one access cycle, N t Indicates the current t The number of accesses to a data block in an access cycle, T Indicates the duration of the unit access cycle, α represents the balance factor; S8: Dynamically adjust the number of copies of each data block in the storage node according to the access frequency of each data block; Specifically, S5 is: to determine the best data block placement strategy through the whale optimization algorithm with the security level as a constraint and the retrieval time as a goal; The inverse of the retrieval time is used as the fitness function of the whale optimization algorithm; Initialize the whale population, which contains multiple whale individuals. The position of each whale individual in the search space represents a data block placement strategy. Generate a random number between 0 and 1 r 1. Calculate the first coefficient A : ; in, a represents the adaptive convergence factor; The adaptive convergence factor a Specifically: ; in, t Indicates the current iteration number, T Indicates the maximum number of iterations; when When , go to the next step; when , enter the random search mechanism; Generate random variables p ; when When , it enters the shrinking and surrounding mechanism; when When the spiral mechanism is entered; Determine the current global optimal position, and other whales move towards the global optimal position, and update their own positions through the shrinking and surrounding mechanism: ; in, X ( t +1) indicates the t + Individual whale positions at iteration 1, X best ( t ) indicates the t The global optimal position at the iteration is A represents the first coefficient, D represents the randomized distance between the individual position of the whale and the current global optimal position, C represents the second coefficient, t Indicates the current iteration number, X ( t ) indicates the t The individual whale position at the iteration; Generate a random number between 0 and 1 r 2. The whale moves in an upward spiral and continuously shrinks the encirclement, updating its position through the spiral mechanism: ; in, represents the distance between the individual position of the whale and the current global optimal position, e represents a natural constant, b represents the logarithmic spiral constant, l Represents a random number between -1 and 1; Based on the positions of individual whales relative to each other, they update their own positions through a random search mechanism: ; in, X rand represents the position of a randomly selected individual whale; Update the global optimal position; Generate a random number r 3. Compare the random number with the mutation probability. If the random number is smaller than the mutation probability, the current position of the individual whale is mutated: ; in, X worst ( t ) indicates the t The global worst position at iteration ; Determine whether the current number of iterations has reached the maximum number of iterations; if so, output the data block placement strategy represented by the whale individual with the highest current fitness as the optimal data block placement strategy; otherwise, return to continue iterating; Wherein, the S8 specifically includes: S801: Obtain the current number of copies of each data block; S802: Calculate the access frequency of the current copies of each data block; S803: Between the minimum number of copies and the maximum number of copies, dynamically adjust the number of copies of each data block in the storage node according to the access frequency of the current existing copies of each data block; The access frequency of the existing copies of the data block is specifically: ; in, F ct Indicates the current t The access frequency of existing copies at the end of the access cycle, F t Indicates the current t The access frequency of the data block at the end of the access cycle, m Indicates the number of existing copies; Wherein, the S803 specifically includes: The 25% data blocks with the highest access frequency among the existing copies are used as hot data blocks, and the number of copies of the hot data blocks is increased; The 25% data blocks with the lowest access frequency of the existing copies are used as non-hotspot data blocks, and the number of copies of the non-hotspot data blocks is reduced; The maximum number of copies is determined as follows: Ensure that the replica update bandwidth is smaller than the system bandwidth: ; in, B update represents the replica update bandwidth, m Represents the number of copies, c u Indicates the size of a single copy. f update Indicates the replica update frequency per unit time; Solution The inequality determines the maximum number of replicas; The method for determining the minimum number of copies is specifically as follows: Get the availability probability expected by the user P a ; When all copies of a data block are unavailable, the data block becomes unavailable. The unavailability probability of each data block is calculated: ; in, P i Indicates i The probability of unavailability of a data block is P ij Indicates i The first data block j The probability of unavailability of a replica, m Indicates the number of copies; When one data block of the target data is unavailable, the entire target data becomes unavailable. The unavailability probability of each target data is calculated: ; in, P f represents the probability of unavailability of target data, n Indicates the total number of data blocks; Ensure availability probability of meeting user expectations: ; in, Indicates from n Select from the elements i The number of combinations of elements, P a Represents the availability probability expected by the user; Solution The inequality determines the minimum number of replicas.

2. The big data storage method according to claim 1, characterized in that: The data block placement strategy specifically includes: a storage node selection sub-strategy and a corresponding data block placement sub-strategy; The storage node selection sub-strategy is specifically: , s i Indicates i Whether a storage node is selected to store data blocks, when When i storage nodes are selected to store data blocks. When i storage nodes are not selected to store data blocks. n Indicates the total number of storage nodes; The data block placement sub-strategy is specifically: , c i Indicates i The size of the data block stored in the storage node is hour, ,and in, C Indicates the size of the data to be stored.

3. The big data storage method according to claim 1, characterized in that: The storage node where the user submits a data read request is the access point p , other storage nodes storing data to be read need to transmit the stored data blocks to the access point using the shortest path transmission method p , the retrieval time is specifically: ; in, Tr Indicates the retrieval time, c i Indicates i The size of the data blocks stored in the storage nodes, n Indicates the total number of storage nodes. L ip Indicates i Storage nodes and access points p The shortest path between i Storage nodes and access points p The shortest path between L ip Including multiple transmission links l , B l Indicates l The bandwidth on the transmission link.

4. The big data storage method according to claim 3, characterized in that: The security level is used to limit the minimum distance between storage nodes storing two data blocks of the same data. The higher the security level, the greater the minimum distance between storage nodes storing two data blocks of the same data.

5. The big data storage method according to claim 4, characterized in that: The S5 is specifically: With the security level as the constraint and the retrieval time as the goal, the best data block placement strategy is determined through the whale optimization algorithm.

6. A big data storage system, characterized in that: include: processor; A memory having computer-readable instructions stored thereon, wherein when the computer-readable instructions are executed by the processor, the big data storage method according to any one of claims 1 to 5 is implemented.