Cloud big data storage management method

Abstract

The invention discloses a cloud big data storage management method, and the method comprises the steps: carrying out the classification of multi-element access habit feature vectors of all terminal nodes through a preset classification model based on the multi-element access habit feature vectors of all terminal nodes, and obtaining a plurality of terminal node groups; based on all the terminal node groups, a first node network and a second node network are constructed, the first node network is used for carrying out distributed storage on the data, and the second node network is used for carrying out distributed backup on the data; monitoring and obtaining target data and a source tracing point thereof in real time, and generating an association processing instruction of the target data according to a preset differentiated source tracing processing decision, the association processing instruction including storage, backup and forwarding; and periodically executing the steps according to a preset time interval. Therefore, based on the dynamic node network, dynamic storage and backup of data are realized, the change of access habits of terminal nodes is adapted, failure storage of nodes in an off-line state is reduced, and the reliability of storage resources is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of data management, and in particular to a cloud big data storage management method. Background Art

[0002] With the rapid development of information network technology, the amount of data stored on the network is becoming increasingly large. Users can upload and download data on their own to save, share and obtain content, which has led to an increasing demand for data storage in the cloud. When users download data and forward it to other users, a copy of the data will be further generated, resulting in a waste of cloud storage resources. The cloud's demand for data backup also further squeezes the remaining cloud storage space, resulting in a low effective utilization rate of cloud storage resources.

[0003] The Chinese invention patent with patent application number 202311237807.2 discloses a cloud-based big data storage management system, which is based on a list of all terminal node devices connected to the cloud, screens terminal nodes based on historical connection cycles, and establishes a cloud storage network; obtains forwarded storage data, and differentiates storage strategies based on data sources: when the data source is the user end, the data is split to generate multiple groups of backup data segments, and appropriate terminal nodes are selected in the cloud storage network for distributed backup storage, and a data storage index table is generated; when the data source is the cloud, the corresponding index link is forwarded.

[0004] However, since the status of several terminal nodes connected to the cloud changes dynamically, the unified construction of the cloud storage network based solely on the historical connection cycle of the simple terminal nodes and the cloud cannot adapt to the changes in the cloud access habits of different user terminals. In addition, since the time points for users to upload data are non-fixed, the time points for data storage and backup are unpredictable. Using a pre-uniformly constructed cloud storage network for distributed storage management leads to efficiency limitations in cloud big data storage management, which can easily cause storage and backup failures of non-online nodes, leading to congestion of cloud storage resources. Summary of the invention

[0005] The present application provides a cloud-based big data storage management method, which realizes dynamic storage and backup of data based on a dynamic node network, adapts to changes in terminal node access habits, reduces failed storage of non-online nodes, and improves the reliability of storage resources.

[0006] This application provides a cloud big data storage management method, including: S101, collecting communication records of all terminal nodes connected to the cloud in all time windows within a preset historical period, the communication records including communication period, communication duration, communication data volume and stability index, and generating a multivariate access habit feature vector for each terminal node; S102, based on the multivariate access habit feature vectors of all terminal nodes, classify them using a preset classification model to obtain a number of terminal node groups, and label each terminal node group, and set the label content as the central feature vector; S103, constructing a first node network and a second node network based on the grouping of all terminal nodes, wherein the first node network is used for distributed storage of data, and the second node network is used for distributed backup of data; S104, real-time monitoring and obtaining of target data and its traceability point, generating associated processing instructions for the target data according to pre-set differentiated traceability processing decisions, the associated processing instructions including storage, backup, and forwarding; S105, periodically executing steps S101 to S104 according to a preset time interval.

[0007] Preferably, generating a multivariate access habit feature vector of each terminal node includes: A1. Obtain the communication time periods in all communication records collected by the terminal node, draw a communication time distribution graph, and obtain the communication time with a density greater than a density threshold to form an active time period; A2. Obtain the communication success rate of each communication data volume as a stability index of data communication; A3. Obtain all communication durations, communication data volumes, and stability indexes in the communication records during the active period, and take the averages to obtain the average communication duration, average communication data volume, and average stability index; A4. Based on the active period, average communication duration, average communication data volume, and average stability index of each terminal node, a multivariate access habit feature vector is constructed.

[0008] Preferably, the preset classification model is set to a K-means clustering algorithm, and S102 specifically includes: All multivariate access habit features are input into the classification model to generate k clustering results. Each cluster corresponds to a terminal node group, and the cluster center of each cluster is used as the label of the corresponding terminal node group.

[0009] Preferably, the method for constructing the first node network includes: retrieving the central feature vector of each terminal node group, selecting the terminal node group whose active period and the current time node meet the adaptation condition as the first target object; wherein the adaptation condition is set as: the current time node is within the active period; constructing the first node network based on all the terminal nodes in the first target object and the cloud; The method for constructing the second node network includes: Retrieve the central feature vector of each terminal node group, select the terminal node group whose stability index is greater than the stability threshold as the second target object, determine the maximum active period coverage combination in the second target object; and construct a second node network based on the terminal nodes and the cloud in all terminal node groups in the maximum active period coverage combination.

[0010] Preferably, according to the preset differentiated traceability processing decision, the associated processing instructions of the target data are generated, specifically including: S201, if the traceability point is a terminal node, use the first node network and the second node network to store and back up the target data respectively, generate index information of the target data and update it to a pre-set cloud index library; the index information includes storage index information and backup index information; S202: If the tracing point is the cloud, retrieve the index information of the target data in the cloud index library and forward it to the target terminal node.

[0011] Preferably, the storing of target data by using the first node network includes: B1. Based on the multivariate access habit feature vectors of all terminal nodes in the first node network, terminal nodes whose active periods meet the adaptation conditions with the current time nodes are selected as distributed storage nodes; B2. Based on the first allocation algorithm, the target data is cut into several data blocks and allocated to corresponding distributed storage nodes for storage; The first allocation algorithm includes: S1. Calculate the size of all data blocks cut from the target data block according to the following formula:

[0012] The size of the data block allocated to each distributed storage node, D is the size of the target data, is the stability index of the i-th distributed storage node, and N is the total number of distributed storage nodes; S2, based on the cutting size of the target data, cutting is performed in sequence according to a preset order to obtain N data blocks, and based on the size of each data block, the data blocks are allocated to the distribution storage nodes corresponding to the size for storage; B3. Label each data block with a unique identifier and record its corresponding distributed storage node to generate an index link. The unique identifiers of all data blocks of the target data and the corresponding index links are determined as the storage index information of the target data. The unique identifier is used to combine all data blocks to obtain the original complete target data.

[0013] Preferably, the backing up of the target data by using the second node network includes: C1. Determine the number of backup sources for the target data based on the number of terminal node groups in the combination with the maximum active period coverage, and each backup source is used to back up the target data once; C2: Based on each backup source, all terminal nodes in it are used as distributed backup nodes; C3. Based on the second allocation algorithm, the target data is cut into several data blocks and allocated to corresponding distributed backup nodes for backup; The second allocation algorithm includes: S3. Calculate the size of all data blocks cut from each backup source using the following formula:

[0014] is the data block size allocated to each distributed backup node in the backup source, D is the size of the target data, is the stability index of the jth distributed backup node in the backup source, and M is the total number of distributed backup nodes in the backup source; S4. For each backup source, based on the cut size corresponding to the target data, the data is divided into M data blocks in a preset order, and each data block is allocated to a distributed backup node corresponding to the size for storage based on the size of the data block; C4. Generate all backup sources of the target data, and mark all data blocks in each backup source with a unique identifier and record the corresponding distributed backup nodes to generate index links. The backup source of the target data, the unique identifiers of all data blocks in the backup source, and the index links corresponding to the data blocks are determined as the backup index information of the target data. The unique identifier is used to combine all data blocks to obtain the original complete target data.

[0015] Preferably, in S2, the method of obtaining the preset order includes: Perform content analysis on the target data and divide it into multiple sub-blocks, so that each sub-block contains a relatively independent data portion; Calculate the important contribution factor of each sub-block to the target data: perform weighted summation of the content proportion of the sub-block in the target data, the business value, and the similarity with the historical uploaded data of its traceability point to obtain the important contribution factor of the sub-block; Obtain the important contribution factor of each sub-block in turn according to the order of the sub-blocks in the target data to form an important contribution factor sequence; Based on the sizes of all elements in the important contribution factor sequence, the comprehensive important trend is obtained, and the direction of the comprehensive important trend is determined as a preset order; Among them, the methods for determining the comprehensive important trends include: The average value of all important contribution factors is obtained as the important threshold; the position of the important contribution factors greater than the important threshold in the important contribution factor sequence is determined. If the position is at the front, the descending order is determined as the comprehensive important trend; if the position is at the back, the ascending order is determined as the comprehensive important trend.

[0016] Preferably, the S2 further includes: S301, determining all cutting points for cutting the target data in a preset order as an initial cutting axis, on which all cutting points are sequentially arranged, and every two adjacent cutting points form a cutting interval segment, and each cutting interval segment corresponds to an initial data block; S302, recording the sub-blocks falling into each cutting interval and their important contribution factors, and calculating the comprehensive important contribution degree of each cutting interval; S303, based on the comprehensive important contribution degree of each cutting interval segment, the initial data block size ratio, and the stability index of the corresponding allocated distribution storage node, a mapping feature vector of the cutting interval segment is generated, and the feature vector is input into a pre-trained mapping relationship abnormality recognition model, and the recognition result of the cutting interval segment is output, the recognition result includes normal and abnormal and their abnormality types, and the abnormality type is set to the comprehensive important contribution degree exceeding the standard and the exceeding proportion, or the comprehensive important contribution degree shortage and the shortage proportion; S304, obtaining an abnormal cutting interval segment based on the recognition result, and using a preset local adjustment strategy to update the size of the initial data block of the abnormal cutting interval segment to a target size, thereby obtaining a target data block; S305, updating the cutting interval segment on the initial cutting point axis of the target data to obtain the target cutting point axis, and cutting the target data according to the target cutting point axis to obtain N data blocks.

[0017] Preferably, the method for acquiring the pre-trained mapping relationship anomaly recognition model includes: E1. Collect the mapping feature vectors of the cut interval segments corresponding to the large number of data blocks assigned to the terminal node in history, and label each historical mapping feature vector based on it, with the label content set as normal, abnormal and its abnormal type; E2. Use the annotated mapping feature vector as a training set, use the training set to train the pre-selected neural network structure, optimize the model parameters, and generate the final mapping relationship anomaly recognition model; The preset local adjustment strategy includes: D1. Obtain the abnormal type of the abnormal cutting interval. If the abnormal type is that the comprehensive important contribution degree exceeds the standard, the first local adjustment algorithm is used to obtain the target size; if the abnormal type is that the comprehensive important contribution degree is insufficient, the second local adjustment algorithm is used to obtain the target size; D2. Dynamically move the reference point of the cutting interval segment so that the size of the initial data block corresponding to the cutting interval segment is adjusted to the target size to obtain the target data block.

[0018] One or more technical solutions provided in this application have at least the following technical effects or advantages: By collecting and analyzing the multivariate access habit feature vectors of terminal nodes, the node network is periodically constructed and adjusted, and the access habits of terminal nodes are analyzed; based on the active period and stability index of the terminal nodes, the first node network and the second node network for data storage and backup are respectively constructed, which ensures the reliability of data backup while efficiently storing data, and improves the dynamic adaptability of storage and backup; real-time monitoring of target data and its traceability points, and generation of associated processing instructions based on differentiated traceability processing decisions, to achieve dynamic storage and backup of data, adapt to changes in terminal node access habits in real time, optimize storage and backup time points, and improve storage resource utilization and data security, reduce failed storage and backup of non-online nodes, and improve storage resource utilization; Traditional data distributed cutting storage adopts a uniform cutting method, without considering the reliability index of the terminal node in the corresponding node network. The first allocation algorithm allocates the data block size according to the stability index of the terminal node in the active period. The second allocation algorithm allocates the data block size in the backup source according to the stability index of the distributed backup node in the corresponding active period. It considers the stability and active period of the terminal nodes in the node network, improves the rationality of data storage and backup, and allocates larger data blocks to terminal nodes with high stability index, improves the efficiency and reliability of data storage and backup, and avoids the waste of resources and imbalance problems that may be caused by uniform allocation. The node network built periodically performs multi-location backup and data storage, effectively solving the problems of insufficient dynamic adaptability and storage resource congestion in traditional cloud storage data. By calculating the important contribution factors of sub-blocks and determining the cutting order according to their comprehensive important trends, the cutting of data blocks is made more reasonable; it avoids to the greatest extent possible that sub-blocks with larger important contribution factors are allocated to terminal nodes with poor stability, thereby reducing storage risks. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 Schematic diagram of the process of the cloud big data storage management method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0020] To facilitate the understanding of the present invention, the present application will be described more comprehensively below with reference to the relevant drawings; the drawings show preferred embodiments of the present invention, but the present invention can be implemented in many different forms and is not limited to the embodiments described herein; on the contrary, the purpose of providing these embodiments is to enable a more thorough and comprehensive understanding of the disclosed content of the present invention.

[0021] It should be noted that the terms “vertical”, “horizontal”, “up”, “down”, “left”, “right” and similar expressions used in this document are only for illustrative purposes and do not represent the only implementation method.

[0022] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by technicians in the technical field to which the present invention belongs; the terms used in the specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention; the term "and / or" used herein includes any and all combinations of one or more related listed items.

[0023] Embodiment 1: Figure 1 It is a flowchart of a cloud big data storage management method according to an embodiment of the present invention.

[0024] like Figure 1 As shown, a cloud big data storage management method includes the following steps: S101, collect the communication records of all terminal nodes connected to the cloud in all time windows within a preset historical period, the communication records include but are not limited to the communication period (connection period), communication duration, communication data volume and its stability index, and generate a multivariate access habit feature vector for each terminal node.

[0025] Among them, the preset historical period is set to the past week, and the time window is set to one day, which can be adjusted according to actual conditions.

[0026] Specifically, a multivariate access habit feature vector is generated for each terminal node, including: A1. Obtain the communication time periods in all communication records collected by the terminal node, draw a communication time distribution diagram, obtain the communication time with a communication frequency greater than a dense threshold, and form an active time period; Wherein, step A1 specifically includes: Obtaining the communication period: Collecting all communication records of the terminal node within the preset historical period, extracting the start and end time of each communication, and forming a communication time series; Draw a communication time distribution graph: Draw the communication time series into a time distribution graph, with the horizontal axis being time (such as hours) and the vertical axis being the communication frequency; Determine the density threshold: According to the communication time distribution graph, set a density threshold to identify the time period with frequent communication activities; Identify active time periods: Traverse the communication time distribution graph, find the communication time with a communication frequency greater than the intensive threshold, and combine these communication times into the active time period of the terminal node.

[0027] For example, in the communication time distribution diagram of a terminal node in the past week, the density threshold is set to 1.5 times the average communication frequency. It is found that the communication frequency of the communication time directly pressed from 9:00 to 11:00 in the morning every day is greater than the density threshold. In this case, the active period of the terminal node is 9:00-11:00 in the morning.

[0028] A2. Obtain the communication success rate of each communication data volume, that is, the integrity of the communication data during the transmission process (the communication data volume received by the receiving end / the communication data volume sent by the sending end, the matching degree of the communication data content sent and the communication data content received, and the weighted sum of the two), to represent the stability index of the terminal node for data communication; A3. Obtain all communication durations, communication data volumes, and stability indexes in the communication records during the active period, and take the averages to obtain the average communication duration, average communication data volume, and average stability index; A4. Based on the active period, average communication duration, average communication data volume, and average stability index of each terminal node, a multivariate access habit feature vector is constructed.

[0029] S102, based on the multivariate access habit feature vectors of all terminal nodes, a preset classification model is used to classify them to obtain a number of terminal node groups, and each terminal node group is labeled, and the labeled content is set as the central feature vector.

[0030] The preset classification model is set to the K-means clustering algorithm, and step S102 specifically includes: All multivariate access habit features are input into the classification model to generate k clustering results. Each cluster corresponds to a terminal node group, and the cluster center (center feature vector) of each cluster is used as the label of the corresponding terminal node group.

[0031] For example, in step 1, the model randomly selects k initial cluster centers, which are points in the multivariate access habit feature space and represent the initial clustering hypothesis; in step 2, there is a terminal node whose multivariate access habit feature vector is [x1, x2, ..., x4], and its distance from each cluster center is calculated, and then it is added to the cluster with the smallest distance; in step 3, after all terminal nodes are assigned to a cluster, the center of each cluster is recalculated (that is, the average value of all terminal node feature vectors in the cluster). For example, for cluster C, its new center feature vector is the mean vector of all terminal node feature vectors in C; in step 4, steps 2 and 3 are repeated until the cluster center no longer changes significantly (that is, the convergence condition is met), or the preset number of iterations is reached. The convergence condition can be that the change of the cluster center is less than a certain threshold, or the clustering result no longer changes; in step 5, finally, each cluster corresponds to a terminal node group, and the center (center feature vector) of each cluster is used as the label of the corresponding terminal node group. Among them, x1 is the active period, x2 is the average communication duration, x3 is the average communication data volume, and x4 is the average stability index.

[0032] It should be noted that the specific classification process of the K-means clustering algorithm can refer to the relevant prior art content, and the present invention will not elaborate on it.

[0033] S103, constructing a first node network and a second node network based on the grouping of all terminal nodes, wherein the first node network is used for distributed storage of data, and the second node network is used for distributed backup of data.

[0034] Specifically, the method for constructing the first node network includes: Retrieve the central feature vector of each terminal node group, and select the terminal node group (at least one) whose active period meets the adaptation condition with the current time node as the first target object; wherein the adaptation condition is set as: the current time node is within the active period; and construct the first node network based on all terminal nodes and the cloud in the first target object.

[0035] Thus, the first node network ensures that data is stored efficiently during active periods.

[0036] Specifically, the method for constructing the second node network includes: Retrieve the central feature vector of each terminal node group, select the terminal node group whose stability index is greater than the stability threshold as the second target object, determine the maximum active period coverage combination in the second target object (the combination of multiple active periods has the largest time share in a time window); build a second node network based on the terminal nodes and the cloud in all terminal node groups in the maximum active period coverage combination.

[0037] Therefore, the second node network ensures the reliability of data backup by selecting stable nodes.

[0038] S104, real-time monitoring and acquisition of target data and its traceability point, generating associated processing instructions for the target data according to pre-set differentiated traceability processing decisions, the associated processing instructions including storage, backup, and forwarding.

[0039] In some embodiments, according to the preset differentiated traceability processing decision, the associated processing instructions of the target data are generated, specifically including: S201, if the traceability point is a terminal node, indicating that the target data is the storage data uploaded by the terminal node, the first node network and the second node network are used to store and back up the target data respectively, generate index information of the target data and update it to the pre-set cloud index library. The index information includes storage index information and backup index information.

[0040] Specifically, using the first node network to store target data includes: B1. Based on the multivariate access habit feature vectors of all terminal nodes in the first node network, terminal nodes whose active periods meet the adaptation conditions with the current time nodes are selected as distributed storage nodes; B2. Based on the first allocation algorithm, the target data is cut into several data blocks and allocated to corresponding distributed storage nodes for storage; The first allocation algorithm includes: S1. Calculate the size of all data blocks cut from the target data block according to the following formula:

[0041] The size of the data block allocated to each distributed storage node, D is the size of the target data, is the stability index of the i-th distributed storage node, and N is the total number of distributed storage nodes.

[0042] S2. Based on the cutting size of the target data (the size of the data block allocated to each distributed storage node), cut the data into N data blocks in a preset order, and store the data in the distributed storage nodes corresponding to the size of each data block.

[0043] B3. Label each data block with a unique identifier and record its corresponding distributed storage node to generate an index link. The unique identifiers of all data blocks of the target data and the corresponding index links are determined as the storage index information of the target data. The unique identifier is used to combine all data blocks to obtain the original complete target data.

[0044] Specifically, backing up the target data using the second node network includes: C1. Based on the number of terminal node groups in the combination with the maximum active period coverage, determine the number of backup sources for the target data. Each terminal node group (i.e., backup source) is used to back up the target data once. C2: Based on each backup source, all terminal nodes in it are used as distributed backup nodes; C3. Based on the second allocation algorithm, the target data is cut into several data blocks and allocated to corresponding distributed backup nodes for backup; The second allocation algorithm includes: S3. Calculate the size of all data blocks cut from each backup source using the following formula:

[0045] is the data block size allocated to each distributed backup node in the backup source, D is the size of the target data, is the stability index of the jth distributed backup node in the backup source, and M is the total number of distributed backup nodes in the backup source.

[0046] S4. For each backup source, based on the cut size corresponding to the target data (the size of the data block allocated to each distributed backup node in the backup source), split it in sequence according to a preset order to obtain M data blocks, and based on the size of each data block, allocate it to the distributed backup node corresponding to the size for storage.

[0047] The preset order of segmentation is the same as the principle in step S2, and the present invention will not elaborate on it.

[0048] C4. Generate all backup sources of the target data, and mark all data blocks in each backup source with a unique identifier and record the corresponding distributed backup nodes to generate index links. The backup source of the target data, the unique identifiers of all data blocks in the backup source, and the index links corresponding to the data blocks are determined as the backup index information of the target data. The unique identifier is used to combine all data blocks to obtain the original complete target data.

[0049] S202, if the tracing point is the cloud, it means that the target data is data that needs to be forwarded from the cloud to the target terminal node, then the index information of the target data in the cloud index library is retrieved and forwarded to the target terminal node.

[0050] Specifically, when the tracing point is the cloud, the storage index information corresponding to the target data in the cloud index library is retrieved, including the unique identifiers of all its data blocks and the corresponding index links, and forwarded to the target terminal node.

[0051] S105, periodically executing steps S101 to S104 at preset time intervals to adapt to changes in access habits of different terminal nodes and avoid limitations of a fixed and unified node network in a dynamic network.

[0052] Among them, the preset time interval is set to 2 hours (online time, active period), and can also be adjusted according to the access situation of the terminal node.

[0053] The technical solutions in the above embodiments of the present application have at least the following technical effects or advantages: By collecting and analyzing the multivariate access habit feature vectors of terminal nodes, the node network is periodically constructed and adjusted, and the access habits of terminal nodes are analyzed; based on the active period and stability index of the terminal nodes, the first node network and the second node network for data storage and backup are respectively constructed, which ensures the reliability of data backup while efficiently storing data, and improves the dynamic adaptability of storage and backup; real-time monitoring of target data and its traceability points, and generation of associated processing instructions based on differentiated traceability processing decisions, to achieve dynamic storage and backup of data, adapt to changes in terminal node access habits in real time, optimize storage and backup time points, and improve storage resource utilization and data security, reduce failed storage and backup of non-online nodes, and improve storage resource utilization; Traditional data distributed cutting storage adopts a uniform cutting method, without considering the reliability index of the terminal node in the corresponding node network. The first allocation algorithm allocates the data block size according to the stability index of the terminal node in the active period. The second allocation algorithm allocates the data block size in the backup source according to the stability index of the distributed backup node in the corresponding active period. It considers the stability and active period of the terminal nodes in the node network, improves the rationality of data storage and backup, and allocates larger data blocks to terminal nodes with high stability index, improves the efficiency and reliability of data storage and backup, and avoids the waste of resources and imbalance problems that may be caused by uniform allocation. Multi-location backup and data storage are performed through a periodically constructed node network, which effectively solves the problems of insufficient dynamic adaptability and storage resource congestion in traditional cloud storage data.

[0054] Embodiment 2: In the traditional scheme, the cutting order of data blocks is fixed or determined based on simple rules. Setting it based only on the order of a single data block size has great limitations. This may be due to ignoring the order of the importance of the target data content, resulting in sub-blocks with larger important contribution factors being allocated to terminal nodes with poor stability, increasing storage burden and hidden dangers.

[0055] Therefore, the embodiments of the present application are optimized to a certain extent based on the above embodiments.

[0056] In some embodiments, in step S2, the method of obtaining the preset order includes: Perform content analysis on the target data and divide it into multiple sub-blocks, so that each sub-block contains a relatively independent data portion; Calculate the important contribution factor of each sub-block to the target data: weighted sum the content proportion of the sub-block in the target data, the business value (which can be set based on expert experience), and the similarity of the historical uploaded data of the corresponding traceability point to obtain the important contribution factor of the sub-block; the weight factor is set according to the actual situation, mainly based on which parameter indicator the important contribution factor depends more on, the corresponding weight factor is larger; Obtain the important contribution factor of each sub-block in turn according to the order of the sub-blocks in the target data (from the beginning to the end) to form an important contribution factor sequence; Based on the size of all elements in the important contribution factor sequence, the comprehensive important trend is obtained, and the direction of the comprehensive important trend is determined as a preset order. The determination method of the comprehensive important trend includes: The average value of all important contribution factors is obtained as the important threshold; the position of the important contribution factors greater than the important threshold in the important contribution factor sequence is determined. If the position is at the front, the descending order is determined as the comprehensive important trend; if the position is at the back, the ascending order is determined as the comprehensive important trend; the judgment standard for being at the front or back is set as: if it is before the middle position of the important contribution factor sequence, it is at the front, otherwise, it is at the back.

[0057] Therefore, the direction of dividing the data block size is the same as the comprehensive importance trend of the important contribution factor, which can avoid the situation where the data block is small but its important contribution factor is large and is allocated to a terminal node with poor stability, thereby increasing the storage burden and hidden dangers of the node. Because the important contribution factor is large and faces more severe and important content protection mechanisms, the node needs higher stability to bear greater storage pressure.

[0058] The technical solutions in the above embodiments of the present application have at least the following technical effects or advantages: By calculating the important contribution factors of sub-blocks and determining the cutting order according to their comprehensive important trends, the cutting of data blocks is made more reasonable; sub-blocks with larger important contribution factors are avoided to the greatest extent possible from being allocated to terminal nodes with poor stability, thereby reducing storage risks; By considering the content proportion, business value and similarity of historical uploaded data of the sub-blocks, the importance of the sub-blocks can be evaluated more accurately, and a sequence of important contribution factors of the target data can be generated according to the importance of the sub-blocks. The cutting order of the target data can be determined based on the comprehensive important trend of the sequence of important contribution factors, so as to reasonably allocate the storage data blocks and improve the storage efficiency and reliability. Based on the setting of the preset order, for sub-blocks with larger important contribution factors, it is ensured that they are allocated to terminal nodes with higher stability, which enhances the data protection mechanism, because data with higher importance requires higher stability to bear greater storage pressure.

[0059] Embodiment 3: In Embodiment 2, the preset order of the cutting size is limited, but it is only roughly determined based on the comprehensive and important trend of the important contribution factors. There must be some mismatches in the data block size, comprehensive importance, and stability index of the corresponding distributed storage nodes in some cutting intervals. This mismatch may cause data blocks with larger important contribution factors to be allocated to nodes with insufficient stability, increasing storage risks and node pressure.

[0060] Therefore, the embodiments of the present application are optimized to a certain extent based on the above embodiments.

[0061] In some embodiments, step S2 further includes: S301, all cutting points for cutting target data in a preset order are determined as an initial cutting axis, all cutting points are sequentially arranged on the initial cutting point axis, every two adjacent cutting points form a cutting interval segment, and each cutting interval segment corresponds to an initial data block.

[0062] S302, recording the sub-blocks and their important contribution factors falling within each cutting interval, and calculating the comprehensive important contribution degree of each cutting interval (taking the average of the important contribution factors of all sub-blocks falling within the cutting interval).

[0063] S303, based on the comprehensive important contribution degree of each cutting interval segment, the initial data block size ratio (the ratio of the initial data block size to the target data size), and the stability index of the corresponding allocated distributed storage node, generate a mapping feature vector of the cutting interval segment, input it into a pre-trained mapping relationship anomaly recognition model, and output the recognition result of the cutting interval segment. The recognition result includes normal and abnormal and their abnormal types. The abnormal type is set to the excess of comprehensive important contribution degree and its excess ratio, and the shortage of comprehensive important contribution degree and its shortage ratio.

[0064] In some embodiments, a method for obtaining a pre-trained mapping relationship anomaly recognition model includes: E1. Collect the mapping feature vectors of the cut interval segments corresponding to the large number of data blocks assigned to the terminal nodes in history, and label each historical mapping feature vector based on it, with the label content set as normal, abnormal and its abnormal type.

[0065] Specifically, the marking method is: Obtain the change range of the stability index of the terminal node within the monitoring period (which can be set to 2 hours and adjusted according to actual conditions) after the data block corresponding to the historical mapping feature vector is allocated to the corresponding terminal node for storage (the calculation method of the stability index can refer to step A2, which is not described in detail in the present invention. The change range of the stability index is obtained based on the stability index of each time node within the monitoring period, and the time node can be set to 5 minutes); If the amplitude of the change in the stability indicator is greater than the preset amplitude threshold, it is marked as abnormal, and its comprehensive important contribution degree and the stability indicator of the corresponding allocated distributed storage node are obtained, and the abnormal type is obtained based on the preset standard rules; wherein the preset standard rules are set as: based on the stability indicator level interval, the corresponding acceptable data block important contribution degree interval is matched, and the level interval and important contribution degree interval of the terminal node in the monitoring period are obtained. If the comprehensive important contribution degree of the historical mapping feature vector is greater than the upper limit value of the important contribution degree interval, the abnormal type is determined to be the comprehensive important contribution degree exceeding the standard and the exceeding proportion (the difference between the comprehensive important contribution degree and the upper limit value of the important contribution degree interval / the upper limit value); if the comprehensive important contribution degree of the historical mapping feature vector is less than the lower limit value of the important contribution degree interval, the abnormal type is determined to be the comprehensive important contribution degree shortage and the shortage proportion (the difference between the comprehensive important contribution degree and the lower limit value of the important contribution degree interval / the lower limit value); otherwise, the label is changed to normal; Otherwise, mark as normal.

[0066] It should be noted that the stability index level interval is pre-set based on experts or historical experience, and the stability index is divided into three level intervals, corresponding to low stability, medium stability, and high stability, and each stability index level interval is matched with an important contribution degree interval of an acceptable data block.

[0067] E2. Use the labeled mapping feature vector as a training set, use the training set to train the pre-selected neural network structure, optimize the model parameters, and generate the final mapping relationship anomaly recognition model.

[0068] S304, obtaining an abnormal cutting interval segment based on the recognition result, and using a preset local adjustment strategy to update the size of the initial data block of the abnormal cutting interval segment to a target size, thereby obtaining a target data block.

[0069] The preset local adjustment strategies include: D1. Obtain the abnormal type of the abnormal cutting interval. If the abnormal type is that the comprehensive important contribution degree exceeds the standard, use the first local adjustment algorithm to obtain the target size; if the abnormal type is that the comprehensive important contribution degree is insufficient, use the second local adjustment algorithm to obtain the target size.

[0070] Among them, when the comprehensive important contribution degree exceeds the standard, it means that when the data block size is adapted to the stability index of the corresponding distributed storage node, the stability index of the corresponding distributed storage node is too low relative to the comprehensive important contribution degree, and the data block size needs to be reduced to reduce the node storage pressure and hidden dangers; when the comprehensive important contribution degree is insufficient, it means that when the data block size is adapted to the stability index of the corresponding distributed storage node, the stability index of the corresponding distributed storage node is too high relative to the comprehensive important contribution degree, and the data block size needs to be increased to fully utilize the node's storage capacity.

[0071] The first local adjustment algorithm is: multiplying the size of the initial data block by the excess ratio to obtain the redundant size, and subtracting the redundant size from the size of the initial data block to obtain the target size; The second local adjustment algorithm is: multiplying the size of the initial data block by the shortage ratio to obtain the supplementary size, and increasing the size of the initial data block by the supplementary size to obtain the target size.

[0072] D2. Dynamically move the reference point of the cutting interval segment (when the abnormal cutting interval segment is located at the beginning and the middle, it is set as the end cutting point of the cutting interval segment; when the abnormal cutting interval segment is located at the end, it is set as the start cutting point of the cutting interval segment), so that the initial data block size corresponding to the cutting interval segment is adjusted to the target size to obtain the target data block.

[0073] S305, updating the cutting interval segment on the initial cutting point axis of the target data to obtain the target cutting point axis, and cutting the target data according to the target cutting point axis to obtain N data blocks.

[0074] It should be noted that the updated abnormal cutting interval segment can be input into the mapping relationship abnormality recognition model to finally check whether the adjustment is successful, or the cutting interval segment in the target cutting point axis can be input into the mapping relationship abnormality recognition model for re-abnormality detection, and steps S304 and S305 can be repeated until there is no abnormal cutting interval segment or the normal rate of the cutting interval segment in the target cutting point axis reaches a preset probability value (which can be set according to actual conditions and experience to limit the overall segmentation reliability), and the loop is stopped.

[0075] The technical solutions in the above embodiments of the present application have at least the following technical effects or advantages: Through the mapping relationship anomaly recognition model, the abnormal results of the cutting interval segment can be accurately identified, providing an accurate basis for subsequent adjustments; according to the type of anomaly, the size of the abnormal cutting interval segment is adjusted using the local adjustment strategy to make the matching of the data block size and the degree of important contribution with the stability index of the corresponding distributed storage node more balanced; through the dynamic adjustment of the data block cutting point position through anomaly detection, it is avoided that the data block is small but its important contribution factor is large when it is allocated to the terminal node with poor stability, thereby improving the storage efficiency and node stability; By introducing the mapping relationship anomaly recognition model and local adjustment strategy, fine-grained adjustment of data block cutting is achieved, which not only improves the rationality of data segmentation, but also enhances the stability and efficiency of data storage; solves the problem of mismatch between data block size, content importance and distributed storage node stability indicators; by dynamically adjusting the data block size, ensures the adaptability of data blocks and node stability indicators, thereby optimizing the allocation of storage resources; improves the reliability and performance of the data storage system; by accurately identifying and processing abnormal cutting intervals, reduces storage risks and node pressure, and improves the reliability and performance of the entire storage system.

[0076] Embodiment 4: In embodiment 1, the node network is periodically updated with a pre-set fixed period, which may not accurately reflect the actual changes in terminal node access habits. The fixed time interval may cause the system to be unable to respond to changes in terminal node access habits in a timely manner, thereby affecting the efficiency and reliability of storage and backup.

[0077] Therefore, the embodiments of the present application are optimized to a certain extent based on the above embodiments.

[0078] In some embodiments, in step S105, the time interval is set as follows: The active period of the central feature vector of all first target objects in the current first node network is obtained, the duration of the minimum active period is selected as the time interval, and S101 to S104 are periodically performed.

[0079] The technical solutions in the above embodiments of the present application have at least the following technical effects or advantages: By selecting the minimum active period duration of the central feature vectors of all first target objects in the first node network as the time interval, it is possible to adapt more dynamically to changes in terminal node access habits, ensuring that the node network can be updated more frequently and reasonably to reflect the latest changes in terminal node access habits, thereby improving the dynamic adaptability of storage and backup, and being able to adapt more quickly to changes in the network environment and the behavior patterns of terminal nodes; it is possible to more accurately determine when to store and back up data, thereby optimizing the time points for storage and backup, which helps to improve the utilization of storage resources and reduce storage and backup failures due to terminal nodes being offline.

[0080] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A cloud big data storage management method, characterized in that: include: S101, collecting communication records of all terminal nodes connected to the cloud in all time windows within a preset historical period, the communication records including communication period, communication duration, communication data volume and stability index, and generating a multivariate access habit feature vector for each terminal node; S102, based on the multivariate access habit feature vectors of all terminal nodes, classify them using a preset classification model to obtain a number of terminal node groups, and label each terminal node group, and set the label content as the central feature vector; S103, constructing a first node network and a second node network based on the grouping of all terminal nodes, wherein the first node network is used for distributed storage of data, and the second node network is used for distributed backup of data; S104, real-time monitoring and obtaining of target data and its traceability point, generating associated processing instructions for the target data according to pre-set differentiated traceability processing decisions, the associated processing instructions including storage, backup, and forwarding; S105, periodically executing steps S101 to S104 according to a preset time interval.

2. The cloud big data storage management method according to claim 1, characterized in that: The step of generating a multivariate access habit feature vector of each terminal node includes: A1. Obtain the communication time periods in all communication records collected by the terminal node, draw a communication time distribution graph, and obtain the communication time with a density greater than a density threshold to form an active time period; A2. Obtain the communication success rate of each communication data volume as a stability index of data communication; A3. Obtain all communication durations, communication data volumes, and stability indexes in the communication records during the active period, and take the averages to obtain the average communication duration, average communication data volume, and average stability index; A4. Based on the active period, average communication duration, average communication data volume, and average stability index of each terminal node, a multivariate access habit feature vector is constructed.

3. The cloud big data storage management method according to claim 2, characterized in that: The preset classification model is set to the K-means clustering algorithm, and S102 specifically includes: All multivariate access habit features are input into the classification model to generate k clustering results. Each cluster corresponds to a terminal node group, and the cluster center of each cluster is used as the label of the corresponding terminal node group.

4. The cloud big data storage management method according to claim 2, characterized in that: The method for constructing the first node network includes: retrieving the central feature vector of each terminal node group, selecting the terminal node group whose active period and the current time node meet the adaptation condition as the first target object; wherein the adaptation condition is set as: the current time node is within the active period; constructing the first node network based on all the terminal nodes in the first target object and the cloud; The method for constructing the second node network includes: Retrieve the central feature vector of each terminal node group, select the terminal node group whose stability index is greater than the stability threshold as the second target object, determine the maximum active period coverage combination in the second target object; and construct a second node network based on the terminal nodes and the cloud in all terminal node groups in the maximum active period coverage combination.

5. The cloud big data storage management method according to claim 4, characterized in that: Generate the associated processing instructions for the target data based on the pre-set differentiated traceability processing decisions, including: S201, if the traceability point is a terminal node, use the first node network and the second node network to store and back up the target data respectively, generate index information of the target data and update it to a pre-set cloud index library; the index information includes storage index information and backup index information; S202: If the tracing point is the cloud, retrieve the index information of the target data in the cloud index library and forward it to the target terminal node.

6. The cloud big data storage management method according to claim 5, characterized in that: The storing of target data by using the first node network includes: B1. Based on the multivariate access habit feature vectors of all terminal nodes in the first node network, terminal nodes whose active periods meet the adaptation conditions with the current time nodes are selected as distributed storage nodes; B2. Based on the first allocation algorithm, the target data is cut into several data blocks and allocated to corresponding distributed storage nodes for storage; The first allocation algorithm includes: S1. Calculate the size of all data blocks cut from the target data block according to the following formula: The size of the data block allocated to each distributed storage node, D is the size of the target data, is the stability index of the i-th distributed storage node, and N is the total number of distributed storage nodes; S2, based on the cutting size of the target data, cutting is performed in sequence according to a preset order to obtain N data blocks, and based on the size of each data block, the data blocks are allocated to the distribution storage nodes corresponding to the size for storage; B3. Label each data block with a unique identifier and record its corresponding distributed storage node to generate an index link. The unique identifiers of all data blocks of the target data and the corresponding index links are determined as the storage index information of the target data. The unique identifier is used to combine all data blocks to obtain the original complete target data.

7. The cloud big data storage management method according to claim 5, characterized in that: The backing up of the target data by using the second node network includes: C1. Determine the number of backup sources for the target data based on the number of terminal node groups in the combination with the maximum active period coverage, and each backup source is used to back up the target data once; C2: Based on each backup source, all terminal nodes in it are used as distributed backup nodes; C3. Based on the second allocation algorithm, the target data is cut into several data blocks and allocated to corresponding distributed backup nodes for backup; The second allocation algorithm includes: S3. Calculate the size of all data blocks cut from each backup source using the following formula: is the data block size allocated to each distributed backup node in the backup source, D is the size of the target data, is the stability index of the jth distributed backup node in the backup source, and M is the total number of distributed backup nodes in the backup source; S4. For each backup source, based on the cut size corresponding to the target data, the data is divided into M data blocks in a preset order, and each data block is allocated to a distributed backup node corresponding to the size for storage based on the size of the data block; C4. Generate all backup sources of the target data, and mark all data blocks in each backup source with a unique identifier and record the corresponding distributed backup nodes to generate index links. The backup source of the target data, the unique identifiers of all data blocks in the backup source, and the index links corresponding to the data blocks are determined as the backup index information of the target data. The unique identifier is used to combine all data blocks to obtain the original complete target data.

8. The cloud big data storage management method according to claim 6, characterized in that: In S2, the method of obtaining the preset order includes: Perform content analysis on the target data and divide it into multiple sub-blocks, so that each sub-block contains a relatively independent data portion; Calculate the important contribution factor of each sub-block to the target data: perform weighted summation of the content proportion of the sub-block in the target data, the business value, and the similarity with the historical uploaded data of its traceability point to obtain the important contribution factor of the sub-block; Obtain the important contribution factor of each sub-block in turn according to the order of the sub-blocks in the target data to form an important contribution factor sequence; Based on the sizes of all elements in the important contribution factor sequence, the comprehensive important trend is obtained, and the direction of the comprehensive important trend is determined as a preset order; Among them, the methods for determining the comprehensive important trends include: The average value of all important contribution factors is obtained as the important threshold; the position of the important contribution factors greater than the important threshold in the important contribution factor sequence is determined. If the position is at the front, the descending order is determined as the comprehensive important trend; if the position is at the back, the ascending order is determined as the comprehensive important trend.

9. The cloud big data storage management method according to claim 8, characterized in that: The S2 further includes: S301, determining all cutting points for cutting the target data in a preset order as an initial cutting axis, on which all cutting points are sequentially arranged, and every two adjacent cutting points form a cutting interval segment, and each cutting interval segment corresponds to an initial data block; S302, recording the sub-blocks falling into each cutting interval and their important contribution factors, and calculating the comprehensive important contribution degree of each cutting interval; S303, based on the comprehensive important contribution degree of each cutting interval segment, the initial data block size ratio, and the stability index of the corresponding allocated distribution storage node, a mapping feature vector of the cutting interval segment is generated, and the feature vector is input into a pre-trained mapping relationship abnormality recognition model, and the recognition result of the cutting interval segment is output, the recognition result includes normal and abnormal and their abnormality types, and the abnormality type is set to the comprehensive important contribution degree exceeding the standard and the exceeding proportion, or the comprehensive important contribution degree shortage and the shortage proportion; S304, obtaining an abnormal cutting interval segment based on the recognition result, and using a preset local adjustment strategy to update the size of the initial data block of the abnormal cutting interval segment to a target size, thereby obtaining a target data block; S305, updating the cutting interval segment on the initial cutting point axis of the target data to obtain the target cutting point axis, and cutting the target data according to the target cutting point axis to obtain N data blocks.

10. The cloud big data storage management method according to claim 9, characterized in that: The method for obtaining the pre-trained mapping relationship anomaly recognition model includes: E1. Collect the mapping feature vectors of the cut interval segments corresponding to the large number of data blocks assigned to the terminal node in history, and label each historical mapping feature vector based on it, with the label content set as normal, abnormal and its abnormal type; E2. Use the annotated mapping feature vector as a training set, use the training set to train the pre-selected neural network structure, optimize the model parameters, and generate the final mapping relationship anomaly recognition model; The preset local adjustment strategy includes: D1. Obtain the abnormal type of the abnormal cutting interval. If the abnormal type is that the comprehensive important contribution degree exceeds the standard, the first local adjustment algorithm is used to obtain the target size; if the abnormal type is that the comprehensive important contribution degree is insufficient, the second local adjustment algorithm is used to obtain the target size; D2. Dynamically move the reference point of the cutting interval segment so that the size of the initial data block corresponding to the cutting interval segment is adjusted to the target size to obtain the target data block.

Images