Data management system and data management method

Abstract

The invention discloses a data management system and a data management method, and relates to the technical field of data processing. According to the method, load parameters of all nodes are monitored in real time, a dynamically updated load state database is established, when new data are collected, a collection time point is taken as a starting point, the load parameters of all the nodes in a set time interval before the starting point are extracted and analyzed to obtain an efficiency index, and the nodes are sorted according to the efficiency index; the first three groups of nodes are selected, the first node is used as a pre-selected node, then the difference value between the efficiency indexes of the second two groups of nodes and the first node is compared, if the difference value is smaller than a preset reference standard value, the node is used as an alternative node, finally the historical fault times of the pre-selected node and the alternative node are compared, and the node with the lower historical fault times is selected as a data transmission target. Therefore, intelligent selection of data transmission nodes is realized, and node loads are balanced.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and in particular to a data management system and a data management method. Background Art

[0002] In today's digital age, the rapid development of information technology has led to explosive growth in various types of data. The sources of data are becoming more extensive, and the forms of data are becoming more diverse, covering structured, semi-structured and unstructured data. These massive amounts of data contain huge potential value, but at the same time they also bring unprecedented challenges to data management.

[0003] However, the data management system and method in the prior art still have the following deficiencies: In terms of data transmission and storage, the data management methods in the existing technology lack accurate perception and intelligent scheduling of node load conditions, and node load imbalance often occurs during data transmission, causing some nodes to have performance degradation due to excessive load, increase data transmission delays, and even cause serious problems such as data loss. In addition, in the data storage process, reasonable optimization has not been carried out based on the actual usage characteristics of the data, resulting in low data storage efficiency and difficulty in meeting the rapidly growing data storage needs.

[0004] To this end, a data management system and data management method are introduced; Summary of the invention

[0005] In view of this, the present invention provides a data management system and a data management method to solve the problems raised by the above background technology.

[0006] The object of the present invention can be achieved by the following technical solution: A data management system comprising: Node selection module: monitors the load parameters of each node in real time, and establishes a dynamically updated load status database based on the real-time monitored load parameters. When new data is collected, the established load status database is retrieved and analyzed for load parameters, and the corresponding node is selected as the data transmission target based on the analysis results; load parameters include CPU usage, memory usage, network bandwidth utilization, network packet loss rate, and task queue length; Data storage module: pre-processes the transmitted new data, stores the transmitted new data in a pre-built pre-evaluation database after the pre-processing is completed, and transfers the data to a pre-built data storage library after the storage lasts for a set period of time; optimizes the layout of the data stored in the data storage library after the preset optimization time interval is reached based on the preset optimization time interval; Data encryption module: uses preset encryption rules to encrypt the data stored in the data repository, and sets the time interval for encryption update. After the set time interval is reached, the preset encryption rules are used again to update the encryption of the stored data.

[0007] In some embodiments, the established load status database is retrieved and analyzed for load parameters, specifically: When new data is collected, the collection time point is taken as the starting point, and the load parameters of each node in the set time interval before the starting point are extracted from the load status database. The CPU usage rate, memory usage rate and bandwidth utilization rate of each node at different time points in the set time interval before the starting point are calculated by averaging, so as to obtain the average CPU usage rate, average memory usage rate and average bandwidth utilization rate of each node in the set time interval before the starting point. The ratio of the number of packet losses of each node to the total number of sent data packets within the set time interval before the statistical starting point is calculated to obtain the network packet loss rate of each node; The average CPU usage, average memory usage, average bandwidth usage, and network packet loss rate of the node are marked as fa, fb, fc, and fd, respectively; at the same time, the reference CPU usage, reference memory usage, reference bandwidth usage, and reference network packet loss rate corresponding to the average CPU usage, average memory usage, average bandwidth usage, and network packet loss rate are preset, and the reference CPU usage, reference memory usage, reference bandwidth usage, and reference network packet loss rate are used , , as well as ; According to the formula The performance evaluation index of each node is obtained by weighted calculation of the average CPU usage fa, average memory usage fb, average bandwidth utilization fc, and network packet loss rate fd of each node. ; Among them, t1, t2, t3 and t4 are the influencing weight factors of the average CPU usage fa, the average memory usage fb, the average bandwidth utilization fc and the network packet loss rate fd respectively.

[0008] In some embodiments, retrieving and analyzing load parameters from the established load status database further includes: Identify the task queue length of each node at the starting point. The task queue length is equal to the number of pending tasks of each node. Identify the difficulty level of the pending tasks of each node, and count the average processing time of different difficulty levels in historical data. For each group of pending tasks of each node, identify the difficulty level and convert the corresponding average processing time. After the conversion is completed, accumulate the average processing time for each group of pending tasks of each node to obtain the required waiting time of each node. Preset the required waiting time for each group of time value ranges, set each group of time value ranges to correspond to an efficiency coefficient; match the required waiting time of each node with each group of time value ranges, so as to obtain the efficiency coefficient corresponding to each node; The efficiency coefficient and performance evaluation index corresponding to each node Multiply them together to get the efficiency index of each node at the current starting point.

[0009] In some embodiments, the corresponding node is selected as the data transmission target according to the analysis result, specifically: According to the size of the efficiency index of each node, they are sorted from large to small, and the nodes in the first three groups are selected from left to right. The first node is used as the pre-selected node, and then the efficiency indexes of the last two groups in the first three groups are extracted, and the difference between them and the efficiency index of the first group is calculated respectively. The reference standard value corresponding to the difference is preset, and the calculated difference is compared with the preset reference standard value. If a group of differences is less than the preset reference standard value, the nodes corresponding to the group of differences are used as candidate nodes, and the historical failure times of the candidate nodes and the pre-selected nodes are extracted from the pre-built historical database and compared, and the nodes with lower historical failure times are selected as the data transmission targets for new data collection.

[0010] In some embodiments, the layout of data stored in the data repository is optimized, specifically: After reaching the preset optimization time interval, a time window for evaluating the data is set with the arrival time point as the starting point. First, the number of accesses to each group of data in the data repository within the time window is counted, and the number of accesses to each group of data is accumulated to obtain the total number of accesses. The number of accesses to each group of data is divided by the total number of accesses to obtain the proportion of the number of accesses to each group of data. At the same time, the duration of each visit corresponding to each set of data is extracted. The duration of the visit starts from when the user retrieves the corresponding data and opens it, and ends when the corresponding data is closed. The duration of each visit to each set of data is accumulated, and the accumulated value is divided by the time window of the evaluation data to obtain the access duration ratio of each set of data; The preset visit times ratio and visit duration ratio correspond to a weight coefficient respectively. The visit times ratio and visit duration ratio of each group of data are multiplied by the corresponding weight coefficient respectively, and then summed up to obtain the heat evaluation index of each group of data.

[0011] In some embodiments, the layout of data stored in the data repository is optimized, specifically: The three groups of index value ranges corresponding to the preset heat evaluation index are set to correspond to a data heat level respectively, and the data heat levels include low heat level, medium heat level and high heat level; Match the heat evaluation index of each set of data with the corresponding index value range to obtain the data heat level of each set of data; A high-speed storage area, a medium storage area and a normal storage area are set in the data storage library, and correspond to a high heat level, a medium heat level and a low heat level respectively; based on the data heat level of each group of data, the storage location of the data is adjusted and optimized.

[0012] In some embodiments, the data stored in the data repository is encrypted using a preset encryption rule, specifically: M1: Use symmetric encryption algorithm to perform basic encryption on the entire data repository; M2: After the basic encryption is completed, a combination code of length m is randomly generated for the data repository, where m>5 and is a positive integer; the combination code is composed of numbers, letters and symbols. After the generation is completed, the letters and symbols in the combination code are converted into numbers using a pre-set coding table, and the coding table sets the numbers corresponding to each group of letters and symbols; the converted numbers are used to replace the letters and symbols in the combination code accordingly; After the replacement is completed, a new string of digital codes is obtained, and a conversion calculation formula is preset, which is an addition, subtraction, multiplication and division formula; Substitute each group of numbers in the new digital code into the preset conversion calculation formula for calculation, and after the calculation is completed, randomly shuffle and sort the calculated groups of numbers; After the sorting is completed, the numbers in the first two groups are extracted and accumulated. The accumulated value is used as the radius of the circle to construct a circle, and the number of remaining numbers is counted as the number of divisions. Starting from the center of the circle, the constructed circle is divided into sub-areas according to the number of divisions. After the division is completed, the remaining numbers are randomly filled into each sub-area. After the filling is completed, the filled circle is used as the encrypted graph of the data storage library.

[0013] In some embodiments, encrypting the data stored in the data repository using a preset encryption rule also includes: M3: After the set time interval is reached, a combination code with a length of m is regenerated for the data storage library, and step M2 is executed for encryption processing.

[0014] In some embodiments, encrypting the data stored in the data repository using a preset encryption rule also includes: M4: Preset the update time interval corresponding to the coding table, and after reaching the update time interval corresponding to the coding table, adjust the numbers corresponding to each group of letters and symbols in the coding table.

[0015] Data management methods, including: Node selection: monitor the load parameters of each node in real time. When new data is collected, analyze the load parameters of each node and select the corresponding node as the data transmission target based on the analysis results. The load parameters include CPU usage, memory usage, network bandwidth utilization, network packet loss rate and task queue length. Storing data: pre-processing the transmitted new data, storing the transmitted new data in a pre-built pre-evaluation database after the pre-processing is completed, and transferring the data to a pre-built data repository after storing it for a set period of time; Storage optimization: based on a preset optimization time interval; after reaching the preset optimization time interval, the layout of the data stored in the data repository is optimized; Data encryption: Use preset encryption rules to encrypt the data stored in the data repository, and set the time interval for encryption update. After the set time interval is reached, use the preset encryption rules again to update the encryption of the stored data.

[0016] Compared with the prior art, the present invention has the following beneficial effects: The present invention establishes a dynamically updated load status database by real-time monitoring of the load parameters of each node. When new data is collected, the collection time point is taken as the starting point, the load parameters of each node within the set time interval before the starting point are extracted and analyzed to obtain the efficiency index, the nodes are sorted according to the efficiency index, the first three groups of nodes are selected, the first ranked node is used as the pre-selected node, and then the difference between the efficiency index of the last two groups of nodes and the first one is compared. If it is less than the preset reference standard value, it is used as an alternative node. Finally, the historical fault times of the pre-selected node and the alternative node are compared, and the node with a lower historical fault time is selected as the data transmission target, thereby realizing the intelligent selection of data transmission nodes and balancing the node load; The present invention is based on a preset optimization time interval. After reaching the preset time interval, the time window for evaluating the data is set with the current time point as the starting point. The proportion of the number of accesses and the proportion of the access duration of each group of data in the data storage library within the time window are counted, and the heat evaluation index is calculated by a preset weight coefficient; the preset heat evaluation index value range corresponds to low, medium and high heat levels, based on which the data storage library is divided into high-speed, medium and ordinary storage areas, and the data storage location is adjusted according to the data heat level, so as to realize the reasonable layout and optimized storage of data and improve storage efficiency; The present invention encrypts the data repository by using preset encryption rules, and presets the code table update time interval. After the expiration, the numbers corresponding to the letters and symbols in the code table are adjusted, and the data encryption security is enhanced through a multi-step, dynamically changing encryption method. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Further details, features and advantages of the present application are disclosed in the following description of exemplary embodiments in conjunction with the accompanying drawings, in which: Figure 1 It is a principle block diagram of the present invention; Figure 2 is a flow chart of the present invention; Figure 3 It is a schematic diagram of the encrypted graph in the present invention. DETAILED DESCRIPTION

[0018] Several embodiments of the present application will be described in more detail below with reference to the accompanying drawings so that those skilled in the art can implement the present application. The present application can be embodied in many different forms and purposes and should not be limited to the embodiments described herein. These embodiments are provided to make the present application comprehensive and complete, and to fully convey the scope of the present application to those skilled in the art. The embodiments do not limit the present application.

[0019] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by those of ordinary skill in the art to which this application belongs. It will be further understood that terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the relevant art and / or the context of this specification, and will not be interpreted in an idealized or overly formal sense unless explicitly defined as such herein.

[0020] Example 1

[0021] See also Figure 1 and Figure 3 As shown, the data management system includes a node selection module, a data storage module and a data encryption module; The node selection module is used to monitor the load parameters of each node in real time, and establish a dynamically updated load status database based on the real-time monitored load parameters. When new data is collected, the established load status database is retrieved and analyzed for load parameters, and the corresponding node is selected as the data transmission target based on the analysis results; the load parameters include CPU usage, memory usage, network bandwidth utilization, network packet loss rate, and task queue length; Specifically: When new data is collected, the collection time point is taken as the starting point, and the load parameters of each node in the set time interval before the starting point are extracted from the load status database. The CPU usage rate, memory usage rate and bandwidth utilization rate of each node at different time points in the set time interval before the starting point are calculated by averaging, so as to obtain the average CPU usage rate, average memory usage rate and average bandwidth utilization rate of each node in the set time interval before the starting point. The ratio of the number of packet losses of each node to the total number of sent data packets within the set time interval before the statistical starting point is calculated to obtain the network packet loss rate of each node; The average CPU usage, average memory usage, average bandwidth usage, and network packet loss rate of the node are marked as fa, fb, fc, and fd, respectively; at the same time, the reference CPU usage, reference memory usage, reference bandwidth usage, and reference network packet loss rate corresponding to the average CPU usage, average memory usage, average bandwidth usage, and network packet loss rate are preset, and the reference CPU usage, reference memory usage, reference bandwidth usage, and reference network packet loss rate are used , , as well as ; According to the formula The performance evaluation index of each node is obtained by weighted calculation of the average CPU usage fa, average memory usage fb, average bandwidth utilization fc, and network packet loss rate fd of each node. ; Among them, t1, t2, t3 and t4 are the influencing weight factors of the average CPU usage fa, the average memory usage fb, the average bandwidth utilization fc and the network packet loss rate fd respectively; Identify the task queue length of each node at the starting point. The task queue length is equal to the number of pending tasks of each node. Identify the difficulty level of the pending tasks of each node, and count the average processing time of different difficulty levels in historical data. The difficulty levels include simple, medium and complex. For each group of pending tasks of each node, identify the difficulty level and convert the corresponding average processing time. After the conversion is completed, accumulate the average processing time of each group of pending tasks of each node to obtain the required waiting time of each node. When processing data, a node usually has a task queue. New data processing tasks will enter the queue and wait for processing. The longer the queue length, the longer the data needs to wait. Preset the required waiting time ranges for each group of time values, set each group of time value ranges to correspond to an efficiency coefficient; the efficiency coefficient value is set to 1.175-1.349; match the required waiting time of each node with the time value range of each group, so as to obtain the efficiency coefficient corresponding to each node; The shorter the waiting time, the higher the efficiency coefficient of the corresponding matching. If the time of a certain node is 0, the value is 1.349; The efficiency coefficient and performance evaluation index corresponding to each node Multiply them together to get the efficiency index of each node at the current starting point; According to the efficiency index size of each node, the nodes are sorted from large to small, and the nodes in the first three groups are selected from left to right. The first node is used as the pre-selected node, and then the efficiency indexes of the last two groups in the first three groups are extracted, and the difference between them and the efficiency index of the first group is calculated respectively, and the reference standard value corresponding to the difference is preset. The calculated difference is compared with the preset reference standard value. If a group of differences is less than the preset reference standard value, the nodes corresponding to the group of differences are used as candidate nodes, and the historical failure times of the candidate nodes and the pre-selected nodes are extracted from the pre-built historical database and compared, and the nodes with the lower historical failure times are selected as the data transmission targets for new data collection; It should be noted that the load of each node is monitored in real time, including indicators such as CPU usage and memory usage; after collecting data, nodes with lighter loads are selected for data transmission based on these load indicators; All nodes are sorted according to the efficiency index, and the node with the highest efficiency index is selected as the transmission target of new data. A high efficiency index means that the node performs well in various load parameters, has a relatively light load, and can process new incoming data more efficiently. If there are multiple nodes with similar efficiency indexes, other factors are further considered to make the final node selection decision; Once the best nodes are determined, the data acquisition device or program can directly transmit the data to the selected nodes via network protocols (such as TCP / IP, UDP, etc.); The data storage module is used to pre-process the transmitted new data; the pre-processing includes cleaning and format conversion, etc.; after the pre-processing is completed, the transmitted new data is stored in a pre-built pre-evaluation database, and after being stored for a set time period, it is transferred to a pre-built data repository; based on a preset optimization time interval; set to one week; after reaching the preset optimization time interval, the layout of the data stored in the data repository is optimized; It should be noted that for new data, it is first stored in the pre-evaluation database by default, and then its accurate heat level is further determined and migrated and adjusted during the subsequent access monitoring process.

[0022] Specifically: After reaching the preset optimization time interval, set a time window for evaluating data with the arrival time point as the starting point; the past month or the past week; first count the number of accesses to each group of data in the data repository within the time window, add up the number of accesses to each group of data to get the total number of accesses, divide the number of accesses to each group of data by the total number of accesses, and thus get the proportion of accesses to each group of data; At the same time, the duration of each visit corresponding to each set of data is extracted. The duration of the visit starts from when the user retrieves the corresponding data and opens it, and ends when the corresponding data is closed. The duration of each visit to each set of data is accumulated, and the accumulated value is divided by the time window of the evaluation data to obtain the access duration ratio of each set of data; A weight coefficient is respectively corresponding to the preset visit times ratio and visit duration ratio, and the visit times ratio and visit duration ratio of each group of data are multiplied by the corresponding weight coefficient respectively, and then summed up to obtain the heat evaluation index of each group of data; The three groups of index value ranges corresponding to the preset heat evaluation index are set to correspond to a data heat level respectively, and the data heat levels include low heat level, medium heat level and high heat level; Match the heat evaluation index of each set of data with the corresponding index value range to obtain the data heat level of each set of data; In the data repository, high-speed storage areas, medium storage areas and ordinary storage areas are set up, corresponding to high heat levels, medium heat levels and low heat levels respectively; based on the data heat level of each group of data, the data storage location is adjusted and optimized; It should be noted that the storage media of the storage areas corresponding to different heat levels are different.

[0023] The data encryption module is used to encrypt the data stored in the data repository using the preset encryption rules, and set the time interval for encryption update. After the set time interval is reached, the encryption of the stored data is updated again using the preset encryption rules; The specific steps are: M1: Basic encryption of the entire data repository using a symmetric encryption algorithm; such as AES; M2: After the basic encryption is completed, a combination code of length m is randomly generated for the data repository, where m>5 and is a positive integer; the numbers in the combination code are all positive integers; the specific values ​​are set by the technical staff; the combination code is composed of numbers, letters and symbols, and after the generation is completed, the letters and symbols in the combination code are converted into numbers using a pre-set coding table, and the numbers corresponding to each group of letters and symbols are set in the coding table; the numbers corresponding to each group of letters and symbols in the coding table are all positive integers; the converted numbers are used to replace the letters and symbols in the combination code; After the replacement is completed, a new string of digital codes is obtained, and a conversion calculation formula is preset, which is an addition, subtraction, multiplication and division formula; the specific preset is made by the technical staff; Substitute each group of numbers in the new digital code into the preset conversion calculation formula for calculation, and after the calculation is completed, randomly shuffle and sort the calculated groups of numbers; It should be noted that when setting the conversion calculation formula, it is necessary to ensure that after the numbers in the digital code are substituted into the conversion calculation formula, the calculated results are all positive integers.

[0024] After the sorting is completed, the numbers in the first two groups are extracted and accumulated, and a circle is constructed using the accumulated value as the radius of the circle, and the number of remaining numbers is counted as the number of divisions. Starting from the center of the circle, the constructed circle is divided into sub-areas according to the number of divisions. After the division is completed, the remaining numbers are randomly filled into each sub-area; after the filling is completed, the filled circle is used as the encrypted graph of the data storage library; For example, randomly generate a combination code of length 6 (A3@b5C); According to the pre-set coding table: "A" corresponds to the number "5"; "@" corresponds to the number "7"; "b" corresponds to the number "9"; "C" corresponds to the number "10"; The digital code obtained after replacement is "5, 3, 7, 9, 5, 10"; The default conversion calculation formula is: (y+3)×2; Substituting the digital code into the conversion formula for calculation, we get 16, 12, 20, 24, 16, 26; After random shuffling, the numbers are 26, 16, 12, 16, 24; Extract the first two groups of numbers "26" and "16" and add them up. 26+16=42, then build a circle with 42 as the radius. The number of remaining numbers is 3, so divide the circle into 3 sub-areas, and randomly fill 16, 12, and 24 into the 3 sub-areas. Finally, use the filled circle as the encrypted graph of the data storage library. M3: after the set time interval is reached, a combination code with a length of m is regenerated for the data storage library, and step M2 is executed for encryption processing; M4: preset the update time interval corresponding to the code table, and after reaching the update time interval corresponding to the code table, adjust the numbers corresponding to each group of letters and symbols in the code table; Example 2

[0025] See also Figure 2 As shown, based on the data management system provided in Example 1 of the present application, Example 2 of the present application proposes a data management method. Example 2 is only a preferred method of Example 1, and the implementation of Example 2 will not affect the independent implementation of Example 1.

[0026] Specifically, the data management method provided in Embodiment 2 of the present application is different in that it includes: Node selection: monitor the load parameters of each node in real time. When new data is collected, analyze the load parameters of each node and select the corresponding node as the data transmission target based on the analysis results. The load parameters include CPU usage, memory usage, network bandwidth utilization, network packet loss rate and task queue length. Storing data: pre-processing the transmitted new data, storing the transmitted new data in a pre-built pre-evaluation database after the pre-processing is completed, and transferring the data to a pre-built data repository after storing it for a set period of time; Storage optimization: based on a preset optimization time interval; after reaching the preset optimization time interval, the layout of the data stored in the data repository is optimized; Data encryption: Encrypt the data stored in the data repository using the preset encryption rules, and set the time interval for encryption update. After the set time interval is reached, the encryption of the stored data is updated using the preset encryption rules again; The preferred embodiments of the present invention disclosed above are only used to help explain the present invention. The preferred embodiments do not describe all the details in detail, nor do they limit the invention to only specific implementation methods. Obviously, many modifications and changes can be made according to the content of this specification. This specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the present invention, so that those skilled in the art can understand and use the present invention well. The present invention is limited only by the claims and their full scope and equivalents.

Claims

1. A data management system, characterized in that: include: Node selection module: monitors the load parameters of each node in real time, and establishes a dynamically updated load status database based on the real-time monitored load parameters. When new data is collected, the established load status database is retrieved and analyzed for load parameters, and the corresponding node is selected as the data transmission target based on the analysis results; Load parameters include CPU usage, memory usage, network bandwidth utilization, network packet loss rate, and task queue length; Data storage module: pre-processes the transmitted new data, stores the transmitted new data in a pre-built pre-evaluation database after the pre-processing is completed, and transfers the data to a pre-built data storage library after the storage lasts for a set period of time; optimizes the layout of the data stored in the data storage library after the preset optimization time interval is reached based on the preset optimization time interval; Data encryption module: uses preset encryption rules to encrypt the data stored in the data repository, and sets the time interval for encryption update. After the set time interval is reached, the preset encryption rules are used again to update the encryption of the stored data.

2. The data management system according to claim 1, characterized in that: The established load status database is used to retrieve and analyze load parameters, specifically: When new data is collected, the collection time point is taken as the starting point, and the load parameters of each node in the set time interval before the starting point are extracted from the load status database. The CPU usage rate, memory usage rate and bandwidth utilization rate of each node at different time points in the set time interval before the starting point are calculated by averaging, so as to obtain the average CPU usage rate, average memory usage rate and average bandwidth utilization rate of each node in the set time interval before the starting point. The ratio of the number of packet losses of each node to the total number of sent data packets within the set time interval before the statistical starting point is calculated to obtain the network packet loss rate of each node; The average CPU usage, average memory usage, average bandwidth usage, and network packet loss rate of the node are marked as fa, fb, fc, and fd, respectively; at the same time, the reference CPU usage, reference memory usage, reference bandwidth usage, and reference network packet loss rate corresponding to the average CPU usage, average memory usage, average bandwidth usage, and network packet loss rate are preset, and the reference CPU usage, reference memory usage, reference bandwidth usage, and reference network packet loss rate are used , , as well as ; According to the formula The performance evaluation index of each node is obtained by weighted calculation of the average CPU usage fa, average memory usage fb, average bandwidth utilization fc, and network packet loss rate fd of each node. ; Among them, t1, t2, t3 and t4 are the influencing weight factors of the average CPU usage fa, the average memory usage fb, the average bandwidth utilization fc and the network packet loss rate fd respectively.

3. The data management system according to claim 2, characterized in that: Retrieval and analysis of load parameters from the established load status database also includes: Identify the task queue length of each node at the starting point. The task queue length is equal to the number of pending tasks of each node. Identify the difficulty level of the pending tasks of each node, and count the average processing time of different difficulty levels in historical data. For each group of pending tasks of each node, identify the difficulty level and convert the corresponding average processing time. After the conversion is completed, accumulate the average processing time for each group of pending tasks of each node to obtain the required waiting time of each node. Preset the required waiting time for each group of time value ranges, set each group of time value ranges to correspond to an efficiency coefficient; match the required waiting time of each node with each group of time value ranges, so as to obtain the efficiency coefficient corresponding to each node; The efficiency coefficient and performance evaluation index corresponding to each node Multiply them together to get the efficiency index of each node at the current starting point.

4. The data management system according to claim 3, characterized in that: According to the analysis results, the corresponding nodes are selected as the data transmission targets, specifically: According to the size of the efficiency index of each node, they are sorted from large to small, and the nodes in the first three groups are selected from left to right. The first node is used as the pre-selected node, and then the efficiency indexes of the last two groups in the first three groups are extracted, and the difference between them and the efficiency index of the first group is calculated respectively. The reference standard value corresponding to the difference is preset, and the calculated difference is compared with the preset reference standard value. If a group of differences is less than the preset reference standard value, the nodes corresponding to the group of differences are used as candidate nodes, and the historical failure times of the candidate nodes and the pre-selected nodes are extracted from the pre-built historical database and compared, and the nodes with lower historical failure times are selected as the data transmission targets for new data collection.

5. The data management system according to claim 4, characterized in that: Optimize the layout of data stored in the data repository, specifically: After reaching the preset optimization time interval, a time window for evaluating the data is set with the arrival time point as the starting point. First, the number of accesses to each group of data in the data repository within the time window is counted, and the number of accesses to each group of data is accumulated to obtain the total number of accesses. The number of accesses to each group of data is divided by the total number of accesses to obtain the proportion of the number of accesses to each group of data. At the same time, the duration of each visit corresponding to each set of data is extracted. The duration of the visit starts from when the user retrieves the corresponding data and opens it, and ends when the corresponding data is closed. The duration of each visit to each set of data is accumulated, and the accumulated value is divided by the time window of the evaluation data to obtain the access duration ratio of each set of data; The preset visit times ratio and visit duration ratio correspond to a weight coefficient respectively. The visit times ratio and visit duration ratio of each group of data are multiplied by the corresponding weight coefficient respectively, and then summed up to obtain the heat evaluation index of each group of data.

6. The data management system according to claim 5, characterized in that: Optimize the layout of data stored in the data repository, specifically: The three groups of index value ranges corresponding to the preset heat evaluation index are set to correspond to a data heat level respectively, and the data heat levels include low heat level, medium heat level and high heat level; Match the heat evaluation index of each set of data with the corresponding index value range to obtain the data heat level of each set of data; A high-speed storage area, a medium storage area and a normal storage area are set in the data storage library, and correspond to a high heat level, a medium heat level and a low heat level respectively; based on the data heat level of each group of data, the storage location of the data is adjusted and optimized.

7. The data management system according to claim 6, characterized in that: The data stored in the data repository is encrypted using the preset encryption rules, specifically: M1: Use symmetric encryption algorithm to perform basic encryption on the entire data repository; M2: After the basic encryption is completed, a combination code of length m is randomly generated for the data repository, where m>5 and is a positive integer; the combination code is composed of numbers, letters and symbols. After the generation is completed, the letters and symbols in the combination code are converted into numbers using a pre-set coding table, and the coding table sets the numbers corresponding to each group of letters and symbols; the converted numbers are used to replace the letters and symbols in the combination code accordingly; After the replacement is completed, a new string of digital codes is obtained, and a conversion calculation formula is preset, which is an addition, subtraction, multiplication and division formula; Substitute each group of numbers in the new digital code into the preset conversion calculation formula for calculation, and after the calculation is completed, randomly shuffle and sort the calculated groups of numbers; After the sorting is completed, the numbers in the first two groups are extracted and accumulated. The accumulated value is used as the radius of the circle to construct a circle, and the number of remaining numbers is counted as the number of divisions. Starting from the center of the circle, the constructed circle is divided into sub-areas according to the number of divisions. After the division is completed, the remaining numbers are randomly filled into each sub-area. After the filling is completed, the filled circle is used as the encrypted graph of the data storage library.

8. The data management system according to claim 7, characterized in that: Encrypting the data stored in the data repository using preset encryption rules also includes: M3: After the set time interval is reached, a combination code with a length of m is regenerated for the data storage library, and step M2 is executed for encryption processing.

9. The data management system according to claim 8, characterized in that: Encrypting the data stored in the data repository using preset encryption rules also includes: M4: Preset the update time interval corresponding to the coding table, and after reaching the update time interval corresponding to the coding table, adjust the numbers corresponding to each group of letters and symbols in the coding table.

10. A data management method, applied to the data management system according to any one of claims 1 to 9, characterized in that: include: Node selection: monitor the load parameters of each node in real time. When new data is collected, analyze the load parameters of each node and select the corresponding node as the data transmission target based on the analysis results. The load parameters include CPU usage, memory usage, network bandwidth utilization, network packet loss rate and task queue length. Storing data: pre-processing the transmitted new data, storing the transmitted new data in a pre-built pre-evaluation database after the pre-processing is completed, and transferring the data to a pre-built data repository after storing it for a set period of time; Storage optimization: based on a preset optimization time interval; after reaching the preset optimization time interval, the layout of the data stored in the data repository is optimized; Data encryption: Use preset encryption rules to encrypt the data stored in the data repository, and set the time interval for encryption update. After the set time interval is reached, use the preset encryption rules again to update the encryption of the stored data.

Images