Mass data collection and storage method and system based on distributed architecture

Abstract

The invention discloses a mass data collection and storage method and system based on a distributed architecture, and relates to the technical field of distributed storage, and the method comprises the following steps: monitoring data traffic in real time, and collecting node resource states; according to the data traffic and the node resource state, the load condition and the data traffic change trend of each current node are analyzed, and according to the analysis result, resource allocation is dynamically coordinated through a distributed architecture; based on the coordinated resource allocation, evaluating the mobility level of the data and classifying the data, and performing adaptive migration and storage on the data according to a classification result; and data mobility changes are monitored in real time, and resource allocation and storage strategies are optimized. According to the method, the access efficiency of high-frequency access data is improved, resource allocation is optimized, and finally the storage efficiency and the overall performance are remarkably improved.

Description

Technical Field

[0001] The present invention relates to the field of distributed storage technology, and in particular to a method and system for collecting and storing massive data based on a distributed architecture. Background Art

[0002] With the rapid development of information technology and the advent of the big data era, the collection, transmission and storage of massive data have become important tasks of modern information systems. Especially with the widespread application of cloud computing and distributed systems, how to efficiently manage massive data has become a focus of attention in all walks of life. Distributed storage systems significantly improve storage capacity and processing capabilities by storing data in multiple nodes, meeting the needs of massive data. However, with the growth of data volume and the diversification of complex business scenarios, how to dynamically adjust resource allocation under a distributed architecture and reasonably store data based on data liquidity has become one of the key issues in current technological development.

[0003] Although existing distributed data collection and storage technologies can achieve the management of massive data to a certain extent, they still have some shortcomings in terms of resource utilization and data storage optimization. First, many existing methods do not fully consider the dynamic changes in data traffic, resulting in relatively static resource allocation strategies that cannot respond to fluctuations in data traffic in real time, which in turn causes some nodes to be overloaded, affecting the overall performance of the system. Secondly, existing storage strategies generally lack real-time monitoring and classification of changes in data liquidity, and are unable to dynamically adjust data storage media and resource allocation based on indicators such as data access frequency and transmission rate, resulting in waste or inefficient use of resources in high-performance storage devices. Summary of the invention

[0004] In view of the above existing problems, the present invention is proposed.

[0005] Therefore, the present invention provides a method and system for collecting and storing massive data based on a distributed architecture to solve the problem in the prior art that resource allocation is static and cannot respond to data flow fluctuations in real time.

[0006] In order to solve the above technical problems, the present invention provides the following technical solutions: In the first aspect, the present invention provides a method for collecting and storing massive data based on a distributed architecture, which includes real-time monitoring of data traffic and collecting node resource status; analyzing the current load conditions of each node and the data traffic change trend based on the data traffic and node resource status, and dynamically coordinating resource allocation through a distributed architecture based on the analysis results; based on the coordinated resource allocation, evaluating the liquidity level of the data and classifying it, and adaptively migrating and storing the data based on the classification results; real-time monitoring of data liquidity changes, and optimizing resource allocation and storage strategies.

[0007] As a preferred solution of the method for collecting and storing massive data based on a distributed architecture described in the present invention, wherein: the data flow includes the data transmission speed, transmission volume, access frequency and data packet size; The node resource status includes CPU usage, memory occupancy, storage space utilization, network bandwidth utilization and I / O read and write load.

[0008] As a preferred solution of the method for collecting and storing massive data based on a distributed architecture described in the present invention, the specific steps of analyzing the current load of each node and the data flow change trend according to the data flow and node resource status are as follows: Based on the CPU usage, memory usage, network bandwidth usage, storage space utilization, and I / O load of each node, and combined with the exponential function and logarithmic function, the comprehensive load index of each node is comprehensively analyzed. The expression is:

[0009] in, represents the CPU usage of the i-th node, represents the memory usage of the i-th node, represents the network bandwidth usage rate of the i-th node, represents the storage space usage of the i-th node, represents the read and write load of the I / O read and write load of node i, represents the comprehensive load index of the i-th node; Define load warning thresholds based on historical performance data ; when ≥ When the current node is set to be in a high load state, < When , the current node is set to be in a low load state; Based on the historical data traffic changes and current resource usage, a trend analysis algorithm is used to predict the data traffic growth rate in the future. The expression is:

[0010] in, Indicates that the i-th node is in the future time period The data traffic growth rate within represents the length of the time window, is the time weight function, is the i-th node at the future time The instantaneous change rate of historical data flow, t represents the current moment, Indicates the future time, Indicates that the i-th node will be in the future The comprehensive load index, is the adjustment coefficient of the current node load on the future data traffic growth rate, Indicates future time The instantaneous rate of change of historical data traffic The weight of the data traffic growth rate in the future period; when When the value is positive, the i-th node in the future time period The data traffic in the country is on the rise; when When the value is negative, the i-th node in the future time period The data traffic in the network is on a downward trend.

[0011] As a preferred solution of the method for collecting and storing massive data based on a distributed architecture described in the present invention, wherein: according to the analysis results, resource allocation is dynamically coordinated through a distributed architecture, and the specific steps are as follows: Based on the distributed architecture, resources are coordinated between different regions according to the load status of each node and the trend of data traffic changes; When a node is in a low-load state, the excess resources of the current node are recovered and allocated to the high-load node; Will be detected in the future time period For nodes with decreasing data traffic, reclaim excess resources in advance and allocate the reclaimed resources to future time periods in advance. Nodes where data traffic is increasing.

[0012] As a preferred solution of the method for collecting and storing massive data based on a distributed architecture described in the present invention, wherein: based on the coordinated resource allocation, the liquidity level of the data is evaluated and classified, and the specific steps are as follows: After dynamically coordinating and allocating resources through a distributed architecture, the data is divided into multiple independent data blocks through a sharding algorithm, and the data block liquidity index is monitored in real time to evaluate the data block liquidity level. The expression is:

[0013] in, represents the liquidity level of the jth data block at time t, represents the transmission rate of the jth data block at time t, represents the access frequency of the jth data block at time t, represents the packet capacity of the jth data block at time t, represents the storage time of the jth database at time t, is the smoothing coefficient, is the nonlinear adjustment coefficient of the data block transmission rate, is the nonlinear adjustment coefficient of the data block access frequency, is the nonlinear adjustment coefficient of the data block inclusion, Indicates the start time At each historical moment between now and the current time t, Indicates the jth data block at the historical moment The historical transmission rate of Indicates the jth data block at the historical moment The historical visit frequency, is the adjustment factor of the data block storage time, is the nonlinear adjustment coefficient of the historical data block transmission rate, is the nonlinear adjustment coefficient of the access frequency of the historical data block; According to the data block liquidity level , the liquidity of each data block is divided into low liquidity level and high liquidity level; Define liquidity standard threshold θ based on historical data and business needs; when ≥θ, the current data block is classified as high liquidity. When θ, the current data block is classified as low liquidity.

[0014] As a preferred solution of the method for collecting and storing massive data based on a distributed architecture described in the present invention, wherein: the data is adaptively migrated and stored according to the classification results, and the specific steps are as follows: According to the liquidity classification, high-liquidity data blocks are migrated to high-performance storage media for storage, and low-liquidity data blocks are migrated to low-cost storage devices for storage, and the migration of high-liquidity data blocks is performed first; After the migration and storage are completed, the storage location index of each data block is updated and the capacity change of the storage device is recorded.

[0015] As a preferred solution of the method for collecting and storing massive data based on a distributed architecture described in the present invention, wherein: the data mobility changes include the dynamic changes of the number of requests, the time interval of access, the transmission rate and the data packet capacity during the migration and storage process; The optimized resource allocation and storage strategy is as follows: When the data liquidity is monitored to continue to increase, the data blocks are migrated to high-performance storage media in advance and resource allocation is increased. When the liquidity continues to decrease, resource allocation is reduced and the data blocks are migrated to low-cost storage devices.

[0016] In the second aspect, the present invention provides a massive data collection and storage system based on a distributed architecture, including a data acquisition module, a resource allocation module, a data storage module and a liquidity optimization module; the data acquisition module is used to monitor data traffic in real time and collect node resource status; the resource allocation module is used to analyze the current load conditions of each node and the data traffic change trend according to the data traffic and node resource status, and dynamically coordinate resource allocation through the distributed architecture based on the analysis results; the data storage module is used to evaluate the liquidity level of the data and classify it based on the coordinated resource allocation, and adaptively migrate and store the data according to the classification results; the liquidity optimization module is used to monitor the changes in data liquidity in real time, and optimize resource allocation and storage strategies.

[0017] In a third aspect, the present invention provides a computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: when the computer program is executed by the processor, it implements any step of the method for collecting and storing massive data based on a distributed architecture as described in the first aspect of the present invention.

[0018] In a fourth aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein: when the computer program is executed by a processor, it implements any step of the method for collecting and storing massive data based on a distributed architecture as described in the first aspect of the present invention.

[0019] The beneficial effects of the present invention are as follows: by real-time monitoring of data traffic and node resource status, dynamic perception of resources is achieved, ensuring that the changing trends of node load and data traffic can be identified in a timely manner, thereby improving responsiveness. Then, by dynamically coordinating resource allocation and adaptively migrating and storing data according to data liquidity, more resources can be allocated to nodes with larger loads, and high-liquidity data can be migrated to high-performance storage media, and low-liquidity data can be migrated to low-cost storage devices. The present invention not only improves the access efficiency of high-frequency access data, but also optimizes resource allocation, ultimately significantly improving storage efficiency and overall performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0021] Figure 1 This is a flow chart of the method for collecting and storing massive data based on a distributed architecture in Example 1.

[0022] Figure 2This is a system diagram of the massive data collection and storage system based on a distributed architecture in Example 1. DETAILED DESCRIPTION

[0023] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are described in detail below in conjunction with the accompanying drawings.

[0024] In the following description, many specific details are set forth to facilitate a full understanding of the present invention, but the present invention may also be implemented in other ways different from those described herein, and those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0025] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The term "in one embodiment" that appears in different places in this specification does not necessarily refer to the same embodiment, nor does it refer to a separate or selective embodiment that is mutually exclusive with other embodiments.

[0026] Example 1, reference Figure 1 and Figure 2 , which is the first embodiment of the present invention, and provides a method for collecting and storing massive data based on a distributed architecture, comprising the following steps: S1: monitors data traffic in real time and collects node resource status; Data traffic includes data transmission speed, transmission volume, access frequency and data packet size; Node resource status includes CPU usage, memory usage, storage space utilization, network bandwidth utilization, and I / O read and write load.

[0027] It should be noted that the transmission speed, transmission volume, access frequency and data packet size of data traffic can be obtained through network protocol stack monitoring tools; resource status such as CPU usage, memory occupancy, storage space utilization, etc. can be obtained through the performance monitoring interface of the operating system (such as Linux's top, iostat and other tools); network bandwidth utilization and I / O read and write load are collected in real time through the monitoring interface of the network and storage devices. These data can reflect the real-time load status and data liquidity of each node, thus providing a basis for dynamic resource allocation and optimization.

[0028] S2: Analyze the current load of each node and the trend of data traffic changes based on data traffic and node resource status. Based on the analysis results, dynamically coordinate resource allocation through a distributed architecture. S2.1: Based on the CPU usage, memory occupancy, network bandwidth usage, storage space utilization and I / O load of each node, combined with the exponential function and logarithmic function, the comprehensive load index of each node is comprehensively analyzed. The expression is:

[0029] in, represents the CPU usage of the i-th node, represents the memory usage of the i-th node, represents the network bandwidth usage rate of the i-th node, represents the storage space usage of the i-th node, represents the read and write load of the I / O read and write load of node i, represents the comprehensive load index of the i-th node; It should be noted that the exponential function and the logarithmic function are used to magnify small changes in node load and compress large numerical differences in comprehensive load analysis. The logarithmic function is used to process the product of multiple resource occupancy values, which can effectively smooth the differences and avoid the impact of a single value that is too large on the overall evaluation; the exponential function amplifies the sudden changes in I / O load and storage space usage, making the response of high node load more sensitive, thereby more accurately measuring the comprehensive load index of each node, which is convenient for dynamic optimization of resource allocation.

[0030] S2.2: Define load warning thresholds based on historical performance data ; when ≥ When the current node is set to be in a high load state, < When , the current node is set to be in a low load state; Furthermore, based on the load warning threshold , dynamically adjust the resource allocation strategy. When a node is in a high-load state, more computing, storage or network resources will be allocated to the node first to prevent performance bottlenecks; when the node is in a low-load state, its excess resources will be recovered and reallocated to high-load nodes or other nodes in need of resources. This dynamic resource allocation mechanism not only improves the overall resource utilization, but also effectively avoids performance degradation or service interruption caused by overload of a single node.

[0031] S2.3: Based on the historical data traffic changes and current resource usage, use the trend analysis algorithm to predict the data traffic growth rate in the future. The expression is:

[0032] in, Indicates that the i-th node is in the future time period The data traffic growth rate within represents the length of the time window, is the time weight function, is the i-th node at the future time The instantaneous change rate of historical data flow, t represents the current moment, Indicates the future time, Indicates that the i-th node will be in the future The comprehensive load index, is the adjustment coefficient of the current node load on the future data traffic growth rate, Indicates future time The instantaneous rate of change of historical data traffic The weight of the data traffic growth rate in the future period; Furthermore, the adjustment coefficient of the current node load on the future data traffic growth rate is The value range of is between [0, 1]. The specific value is adjusted through experiments or experience based on the relationship between historical load and traffic growth to balance the impact of node load on traffic prediction and ensure accuracy and stability under different load conditions.

[0033] r is the time weight function, which is obtained by weighted moving average based on experience or historical data. The expression is: in, is the attenuation coefficient; The instantaneous rate of change is obtained by calculating the rate of change of a variable (such as data traffic) in a very short period of time. The specific method is: first, collect traffic data of each node at different time points; then calculate the traffic change between adjacent time points; finally, divide the change by the time interval to get the instantaneous rate of change. This process can reflect the immediate increase or decrease trend of data traffic in the time period and provide a basis for predicting future traffic changes.

[0034] The trend analysis algorithm predicts future trends by analyzing historical data and current status. Based on the historical data change rate and current node load conditions, combined with time weight, it calculates the traffic growth or decline trend in the future.

[0035] when When the value is positive, the i-th node in the future time period The data traffic in the country is on the rise; when When the value is negative, the i-th node in the future time period The data traffic in the network is on a downward trend.

[0036] Furthermore, through the mechanism of allocating resources in advance, performance bottlenecks or resource waste caused by improper resource allocation can be effectively avoided, and the overall resource utilization efficiency and responsiveness can be improved.

[0037] S2.4: Based on the distributed architecture, resources are coordinated between different regions according to the load status of each node and the trend of data traffic changes; When a node is in a low-load state, the excess resources of the current node are recovered and allocated to the high-load node; Will be detected in the future time period For nodes with decreasing data traffic, reclaim excess resources in advance and allocate the reclaimed resources to future time periods in advance. Nodes where data traffic is increasing.

[0038] Furthermore, redundant resources refer to computing, storage, network bandwidth and other resources that are not fully utilized by the node under low load conditions, for example, idle CPU, memory or unoccupied bandwidth.

[0039] It should also be noted that the resource coordination mechanism is based on a distributed architecture and dynamically adjusts resource allocation by monitoring the load status and data traffic change trend of each node. When a node is under low load or the traffic is expected to decrease in the future, its excess resources are recovered in advance and reallocated to nodes with high load or expected traffic growth. This mechanism ensures the reasonable scheduling of resources between different nodes, avoids resource waste, and improves the performance of high-load nodes, ultimately achieving the optimization of overall resource utilization.

[0040] S3: Based on the coordinated resource allocation, the data mobility level is evaluated and classified, and the data is adaptively migrated and stored according to the classification results; S3.1: After dynamically coordinating and allocating resources through a distributed architecture, the data is divided into multiple independent data blocks through a sharding algorithm, and the data block liquidity index is monitored in real time to evaluate the data block liquidity level. The expression is:

[0041] in, represents the liquidity level of the jth data block at time t, represents the transmission rate of the jth data block at time t, represents the access frequency of the jth data block at time t, represents the packet capacity of the jth data block at time t, represents the storage time of the jth database at time t, is the smoothing coefficient, is the nonlinear adjustment coefficient of the data block transmission rate, is the nonlinear adjustment coefficient of the data block access frequency, is the nonlinear adjustment coefficient of the data block inclusion, Indicates the start time At each historical moment between now and the current time t, Indicates the jth data block at the historical moment The historical transmission rate of Indicates the jth data block at the historical moment The historical visit frequency, is the adjustment factor of the data block storage time, is the nonlinear adjustment coefficient of the historical data block transmission rate, is the nonlinear adjustment coefficient of the access frequency of the historical data block; According to the data block liquidity level , the liquidity of each data block is divided into low liquidity level and high liquidity level.

[0042] Furthermore, the expression comprehensively evaluates the liquidity level of data blocks through multiple key indicators (such as transmission rate, access frequency, packet capacity, etc.), and introduces nonlinear adjustment coefficients (β and δ) to flexibly adjust the impact of different indicators on liquidity. In addition, the historical behavior and current status of data blocks are dynamically captured by using exponential functions, logarithmic functions and weighted integration of historical periods, so as to more accurately evaluate the liquidity of data blocks. This method ensures that storage and resource allocation can be adaptively adjusted according to the flow characteristics of data blocks, improving storage efficiency and response speed.

[0043] The value range of the nonlinear adjustment coefficient is between [0, 1], depending on the design requirements and business scenarios, and is adjusted through experiments to balance the impact of different indicators on data fluidity.

[0044] S3.2: Define the liquidity standard threshold θ based on historical data and business needs; when ≥θ, the current data block is classified as high liquidity. When θ, the current data block is classified as low liquidity.

[0045] Furthermore, different liquidity levels will correspond to different resource requirements. High-liquidity nodes require more bandwidth resources and computing resources, while low-liquidity nodes can appropriately reduce resource allocation.

[0046] S3.3: Based on the liquidity classification, high-liquidity data blocks are migrated to high-performance storage media for storage, and low-liquidity data blocks are migrated to low-cost storage devices for storage, and the migration of high-liquidity data blocks is performed first; After the migration and storage are completed, the storage location index of each data block is updated and the capacity change of the storage device is recorded.

[0047] It should be noted that by classifying data blocks according to their liquidity and migrating high-liquidity data blocks to high-performance storage media (such as SSD or memory cache) first, the reading speed of high-frequency access data can be significantly improved; at the same time, migrating low-liquidity data blocks to low-cost storage devices (such as HDD or remote storage) can effectively save high-performance storage resources. This process ensures high efficiency in resource allocation and maintains accuracy and consistency in data management by updating storage location indexes and recording capacity changes.

[0048] S4: Monitor data liquidity changes in real time and optimize resource allocation and storage strategies.

[0049] Data mobility changes include the number of requests, access time intervals, transmission rates, and dynamic changes in data packet capacity during migration and storage; The optimization resource allocation and storage strategies are as follows: When the data liquidity is monitored to continue to increase, the data blocks are migrated to high-performance storage media in advance and resource allocation is increased. When the liquidity continues to decrease, resource allocation is reduced and the data blocks are migrated to low-cost storage devices.

[0050] It should be noted that by real-time monitoring of data access frequency, time interval, transmission rate, data packet capacity and other liquidity changes, resource allocation and storage location are dynamically adjusted. When data liquidity increases, data is migrated to high-performance storage media in advance and resource investment is increased to improve access efficiency; when liquidity decreases, resource allocation is reduced and data is migrated to low-cost storage devices, thereby maximizing resource utilization and optimizing storage costs.

[0051] This embodiment also provides a system for collecting and storing massive data based on a distributed architecture, including: a data collection module, a resource allocation module, a data storage module and a liquidity optimization module; Data collection module, used to monitor data traffic in real time and collect node resource status; The resource allocation module is used to analyze the current load of each node and the trend of data traffic changes according to the data traffic and node resource status, and dynamically coordinate resource allocation through the distributed architecture based on the analysis results; A data storage module, which is used to evaluate the liquidity level of data and classify it based on the coordinated resource allocation, and adaptively migrate and store the data according to the classification results; The liquidity optimization module is used to monitor data liquidity changes in real time and optimize resource allocation and storage strategies.

[0052] This embodiment also provides a computer device, which is suitable for the case of a method for collecting and storing massive data based on a distributed architecture, including: a memory and a processor; the memory is used to store computer-executable instructions, and the processor is used to execute computer-executable instructions, thereby realizing the method for collecting and storing massive data based on a distributed architecture as proposed in the above embodiment.

[0053] The computer device may be a terminal, and the computer device includes a processor, a memory, a communication interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be achieved through WIFI, an operator network, NFC (near field communication) or other technologies. The display screen of the computer device may be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device may be a touch layer covered on the display screen, or a key, trackball or touchpad provided on the housing of the computer device, or an external keyboard, touchpad or mouse, etc.

[0054] This embodiment also provides a storage medium on which a computer program is stored. When the program is executed by a processor, the method for collecting and storing massive data based on a distributed architecture as proposed in the above embodiment is implemented; the storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, disk or optical disk.

[0055] In summary, the present invention achieves dynamic perception of resources by: real-time monitoring of data traffic and node resource status, ensuring that changing trends of node load and data traffic can be identified in a timely manner, thereby improving responsiveness. Then, by dynamically coordinating resource allocation and adaptively migrating and storing data according to data liquidity, more resources can be allocated to nodes with larger loads, and high-liquidity data can be migrated to high-performance storage media, and low-liquidity data can be migrated to low-cost storage devices. The present invention not only improves the access efficiency of high-frequency access data, but also optimizes resource allocation, ultimately significantly improving storage efficiency and overall performance.

[0056] Example 2, referring to Table 1, is the second example of the present invention. To further verify the technical solution of the present invention, experimental simulation data of a method for collecting and storing massive data based on a distributed architecture is provided.

[0057] In order to verify the advantages of the present invention in the dynamic coordination of data resources and storage strategy optimization, an experiment was conducted on node load analysis, data liquidity evaluation, resource allocation and storage optimization under a distributed architecture. Four nodes were used in the experiment to simulate different load and data liquidity conditions.

[0058] First, during the experimental preparation stage, performance monitoring tools and network protocol stack monitoring tools are used to collect real-time data traffic (including transmission speed, transmission volume, access frequency, and packet size) and node resource status (including CPU utilization, memory occupancy, storage space utilization, network bandwidth utilization, and I / O read and write load) of each node in order to provide basic data for subsequent load analysis and resource optimization.

[0059] Secondly, using the collected node resources and data flow information, a mathematical formula is used to calculate the comprehensive load index of each node, and the load warning threshold is defined based on historical data. When the load index of a node exceeds the warning threshold, the node is automatically identified as being in a high-load state and more resources are allocated to it first; on the contrary, the excess resources of low-load nodes will be recycled and reallocated.

[0060] Then, through the trend analysis algorithm, combined with the time weight function and historical traffic data, the data traffic change trend of each node in the future is predicted. For nodes with obvious traffic growth trends, more resources are allocated to them in advance, and high-mobility data blocks are migrated to high-performance storage media; for nodes with expected traffic decline, resource allocation is reduced, and low-mobility data blocks are migrated to low-cost storage devices.

[0061] Finally, adaptive migration and storage optimization are performed based on the liquidity level of data blocks to ensure that high-liquidity data blocks are stored on high-performance devices first, improving data access efficiency. At the same time, low-performance storage devices are used reasonably to store low-liquidity data, thereby maximizing resource utilization and optimizing costs.

[0062] The details are shown in Table 1 below:

[0063] Through the analysis of the data in the above table, it can be clearly seen that the present invention has significant advantages in real-time monitoring of node load, predicting data traffic changes and dynamically adjusting resource allocation. For example, the CPU utilization rate of node 1 reaches 82.4%, the memory occupancy rate is 74.9%, the network bandwidth utilization rate is 71.3%, and the comprehensive load index is 5.27, reflecting that the node is in a high load state. At the same time, the liquidity level of node 1 is 0.84, and the predicted traffic growth rate is 0.22, indicating that the data traffic of the node will continue to increase in the future. Therefore, more resources are allocated to node 1, and high-liquidity data blocks are migrated to high-performance storage devices (such as SSDs) in advance to cope with the upcoming high-load access needs.

[0064] Through the dynamic coordination of resource allocation of the present invention, resource allocation and storage strategies can be accurately adjusted according to the current load of the node and future traffic trends, which not only avoids resource waste but also effectively improves the access performance of high-load nodes, fully demonstrating the practicality of the present invention in resource management and efficiency improvement.

[0065] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention, which should all be included in the scope of the claims of the present invention.

Claims

1. A method for collecting and storing massive data based on a distributed architecture, characterized in that: Including,real-time monitoring of data traffic and collecting node resource status; According to the data flow and node resource status, analyze the current load of each node and the trend of data flow changes, and dynamically coordinate resource allocation through the distributed architecture based on the analysis results; Based on the coordinated resource allocation, the liquidity level of data is evaluated and classified, and the data is adaptively migrated and stored according to the classification results; Monitor data liquidity changes in real time and optimize resource allocation and storage strategies; According to the data flow and node resource status, the current load situation of each node and the data flow change trend are analyzed. The specific steps are as follows: Based on the CPU usage, memory usage, network bandwidth usage, storage space utilization and I / O load of each node, combined with exponential and logarithmic functions, the comprehensive load index of each node is comprehensively analyzed. ; Define load warning thresholds based on historical performance data ; when ≥ When the current node is set to be in a high load state, < When , the current node is set to be in a low load state; Based on historical data traffic changes and current resource usage, a trend analysis algorithm is used to predict the data traffic growth rate in the future. ; when When the value is positive, the i-th node in the future time period The data traffic in the country is on the rise; when When the value is negative, the i-th node in the future time period The data traffic in the network is on a downward trend.

2. The method for collecting and storing massive data based on a distributed architecture as claimed in claim 1, characterized in that: The data traffic includes data transmission speed, transmission volume, access frequency and data packet size; The node resource status includes CPU usage, memory occupancy, storage space utilization, network bandwidth utilization and I / O read and write load.

3. The method for collecting and storing massive data based on a distributed architecture as claimed in claim 2, characterized in that: Comprehensive load index The expression is: in, represents the CPU usage of the i-th node, represents the memory usage of the i-th node, represents the network bandwidth usage rate of the i-th node, represents the storage space usage of the i-th node, represents the read and write load of the I / O read and write load of node i, represents the comprehensive load index of the i-th node; Data traffic growth rate The expression is: in, Indicates that the i-th node is in the future time period The data traffic growth rate within represents the length of the time window, is the time weight function, is the i-th node at the future time The instantaneous change rate of historical data flow, t represents the current moment, Indicates the future time, Indicates that the i-th node will be in the future The comprehensive load index, is the adjustment coefficient of the current node load on the future data traffic growth rate, Indicates future time The instantaneous rate of change of historical data traffic The weight of the data traffic growth rate in the future period.

4. The method for collecting and storing massive data based on a distributed architecture as claimed in claim 3, characterized in that: According to the analysis results, resource allocation is dynamically coordinated through a distributed architecture. The specific steps are as follows: Based on the distributed architecture, resources are coordinated between different regions according to the load status of each node and the trend of data traffic changes; When a node is in a low-load state, the excess resources of the current node are recovered and allocated to the high-load node; Will be detected in the future time period For nodes with decreasing data traffic, reclaim excess resources in advance and allocate the reclaimed resources to future time periods in advance. Nodes where data traffic is increasing.

5. The method for collecting and storing massive data based on a distributed architecture as claimed in claim 4, characterized in that: The steps of evaluating and classifying the liquidity level of data based on the coordinated resource allocation are as follows: After dynamically coordinating and allocating resources through a distributed architecture, the data is divided into multiple independent data blocks through a sharding algorithm, and the data block liquidity index is monitored in real time to evaluate the data block liquidity level. The expression is: in, represents the liquidity level of the jth data block at time t, represents the transmission rate of the jth data block at time t, represents the access frequency of the jth data block at time t, represents the packet capacity of the jth data block at time t, represents the storage time of the jth database at time t, is the smoothing coefficient, is the nonlinear adjustment coefficient of the data block transmission rate, is the nonlinear adjustment coefficient of the data block access frequency, is the nonlinear adjustment coefficient of the data block inclusion, Indicates the start time At each historical moment between now and the current time t, Indicates the jth data block at the historical moment The historical transmission rate of Indicates the jth data block at the historical moment The historical visit frequency, is the adjustment coefficient of the data block storage time, is the nonlinear adjustment coefficient of the historical data block transmission rate, is the nonlinear adjustment coefficient of the access frequency of the historical data block; According to the data block liquidity level , the liquidity of each data block is divided into low liquidity level and high liquidity level; Define liquidity standard threshold θ based on historical data and business needs; when ≥θ, the current data block is classified as high liquidity. When θ, the current data block is classified as low liquidity.

6. The method for collecting and storing massive data based on a distributed architecture as claimed in claim 5, characterized in that: The specific steps of adaptively migrating and storing data according to the classification results are as follows: According to the liquidity classification, high-liquidity data blocks are migrated to high-performance storage media for storage, and low-liquidity data blocks are migrated to low-cost storage devices for storage, and the migration of high-liquidity data blocks is performed first; After the migration and storage are completed, the storage location index of each data block is updated and the capacity change of the storage device is recorded.

7. The method for collecting and storing massive amounts of data based on a distributed architecture as claimed in claim 6, characterized in that: The data mobility changes include the dynamic changes in the number of requests, access time intervals, transmission rates and data packet capacity during the migration and storage process; The optimized resource allocation and storage strategy is as follows: When the data liquidity is monitored to continue to increase, the data blocks are migrated to high-performance storage media in advance and resource allocation is increased. When the liquidity continues to decrease, resource allocation is reduced and the data blocks are migrated to low-cost storage devices.

8. A system for collecting and storing massive data based on a distributed architecture, based on the method for collecting and storing massive data based on a distributed architecture according to any one of claims 1 to 7, characterized in that: Including data collection module, resource allocation module, data storage module and liquidity optimization module; Data collection module, used to monitor data traffic in real time and collect node resource status; The resource allocation module is used to analyze the current load of each node and the trend of data traffic changes according to the data traffic and node resource status, and dynamically coordinate resource allocation through the distributed architecture based on the analysis results; A data storage module, which is used to evaluate the liquidity level of data and classify it based on the coordinated resource allocation, and adaptively migrate and store the data according to the classification results; The liquidity optimization module is used to monitor data liquidity changes in real time and optimize resource allocation and storage strategies.

Images