Distributed file data processing method and system based on cloud computing and storage medium

Abstract

The invention relates to the technical field of data processing, in particular to a distributed file data processing method and system based on cloud computing and a storage medium, the method comprises the following steps: S1, acquiring basic data of each distributed storage node, processing to obtain a comprehensive performance evaluation value of each distributed storage node, and screening the nodes to obtain each target node; s2, the file processing system writes data and divides the data into blocks, the blocks are marked as data blocks, and the data blocks are copied to the target nodes; s3, collecting system backup data, and processing the system backup data to obtain a backup efficiency evaluation value of each data block; and S4, performing comprehensive analysis to obtain a comprehensive energy efficiency evaluation index of each data block backup, evaluating the backup quality of each data block, and recording an evaluation result. By providing the distributed file data processing method and system based on cloud computing and the storage medium, it is helped to ensure that node performance is matched with backup efficiency, the backup quality is improved, and the backup efficiency is improved. And efficient operation of the backup system is ensured.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a distributed file data processing method, system and storage medium based on cloud computing. Background Art

[0002] With the widespread popularity of the Internet, the amount of data generated in people's daily lives has exploded. These data cover various fields, such as social media, e-commerce, and the Internet of Things. Traditional data processing methods can no longer meet the needs of such large-scale data processing. Cloud computing, as an Internet-based computing resource sharing and distributed processing model, has emerged.

[0003] For example, the invention patent with announcement number: CN105589887B is a data processing method and a distributed file system of a distributed file system, and the method includes: Fac obtains file data and pushes it to Fas; Fas records the file data pushed by Fac, records the corresponding metadata modification on Fas in the buffer, writes it to the log file, and returns the file data push completion message to Fac; Fac sends a metadata modification change request to Flr; Flr modifies the corresponding metadata according to the metadata modification change request and records it to the log file system; when it is detected that Fas restarts abnormally, Flr performs a rollback operation on the corresponding modified data according to the log record.

[0004] For example, the invention patent with announcement number: CN118484432B is a distributed file data processing method and device based on cloud computing, which belongs to the field of data processing technology. By using a distributed storage system and computing nodes, and configuring data mining models and subtasks, the data mining model can automatically decompose tasks into subtasks, and allocate data blocks according to dependencies and rules. It can flexibly adjust the data processing flow and mining sub-algorithms according to specific needs, adapt to different processing tasks and complex data types, process large amounts of data in parallel, save storage space, improve the efficiency and speed of data processing, and then integrate and display the target data processing results. It can intuitively present the effects and results of data processing and provide users with useful information.

[0005] However, in the process of implementing the technical solution of the invention in the embodiments of the present application, the present application discovered that the above-mentioned technology has at least the following technical problems: the existing processing methods often adopt a distributed storage method, which may cause errors or loss of data during transmission and synchronization, thereby affecting the integrity and accuracy of the data. Summary of the invention

[0006] In view of the deficiencies in the prior art, the present invention provides a distributed file data processing method, system and storage medium based on cloud computing, which can effectively solve the problems involved in the above-mentioned background technology.

[0007] To achieve the above objectives, the present invention is implemented through the following technical solutions: The first aspect of the present invention provides a distributed file data processing method based on cloud computing, including: S1. Acquire the basic data of each distributed storage node, process it to obtain the comprehensive performance evaluation value of each distributed storage node, and screen the nodes according to the comprehensive performance evaluation value of each distributed storage node to obtain each target node.

[0008] S2. The file processing system writes the data and divides the data into blocks, marks them into data blocks, and copies each data block to each target node.

[0009] S3. Collect system backup data and obtain the backup efficiency evaluation value of each data block after processing.

[0010] S4. According to the comprehensive performance evaluation value of each target node and the backup efficiency evaluation value of each data block, a comprehensive analysis is performed to obtain the comprehensive energy efficiency evaluation index of each data block backup, and the backup quality of each data block is evaluated according to the comprehensive energy efficiency evaluation index of each data block backup, and the evaluation results are recorded.

[0011] As a further method, the basic data of each distributed storage node specifically includes: node storage performance data, node network performance data, node average uptime and node resource utilization.

[0012] The node stores performance data, including throughput and read and write speeds at each monitoring point in time.

[0013] The node network performance data includes network delay and network bandwidth at each time monitoring point.

[0014] As a further method, the processing obtains the comprehensive performance evaluation value of each distributed storage node. The specific process is: extracting the critical node normal operation time, critical node resource utilization, critical node throughput, critical node read and write speed, critical node network delay and critical node network bandwidth from the data processing system database.

[0015] Based on the basic data of each distributed storage node, a comprehensive analysis is performed to obtain a comprehensive performance evaluation value of each distributed storage node. The comprehensive performance evaluation value of each distributed storage node is used to quantitatively evaluate the reliability of each distributed storage node in performing data backup, providing a basis for node screening.

[0016] As a further method, the nodes are screened to obtain target nodes. The specific screening process is: extract the comprehensive performance evaluation threshold of the distributed storage nodes from the data processing system database, compare the comprehensive performance evaluation value of each distributed storage node with the comprehensive performance evaluation threshold of the distributed storage node, if the comprehensive performance evaluation value of a distributed storage node is higher than or equal to the comprehensive performance evaluation threshold of the distributed storage node, then the performance evaluation of the distributed storage node is qualified and marked as a target node, if the comprehensive performance evaluation value of a distributed storage node is lower than the comprehensive performance evaluation threshold of the distributed storage node, then the performance evaluation of the distributed storage node is unqualified.

[0017] Count the target nodes.

[0018] As a further method, the data blocks are copied to the target nodes. The specific process is as follows: U1. The file processing system receives a data write request from the user and temporarily stores the data in the system buffer.

[0019] U2. Cut the temporarily stored data into pieces according to the preset block size, generate data blocks, and mark them as data blocks.

[0020] U3. Match the target node for each data block according to the system's load balancing algorithm and establish a network connection with the target node.

[0021] U4. Transmit each data block in parallel to the corresponding target node, and mark the node storing the data block as each storage node for each data block.

[0022] As a further method, the backup efficiency evaluation value of each data block is obtained through processing, and the specific process is: the system backs up data, including the total backup time, average transmission rate, backup success rate and fault recovery time of each data block.

[0023] The total backup time of critical data blocks, the average transmission rate of critical data blocks, the backup success rate of critical data blocks and the failure recovery time of critical data blocks are extracted from the data processing system database.

[0024] According to the system backup data, the backup efficiency evaluation value of each data block is processed to obtain the backup efficiency evaluation value of each data block, which is used to quantitatively evaluate the backup efficiency of each data block and provide a basis for quality evaluation of the data block.

[0025] As a further method, the comprehensive analysis obtains the comprehensive energy efficiency evaluation index of each data block backup. The specific analysis process is: taking the average value of the comprehensive performance evaluation value of each storage node corresponding to each data block to obtain the comprehensive performance evaluation value of each data block storage node.

[0026] According to the comprehensive performance evaluation value of each data block storage node and the evaluation value of each data block backup efficiency, a comprehensive analysis is performed to obtain a comprehensive energy efficiency evaluation index for each data block backup. The comprehensive energy efficiency evaluation index for each data block backup is used to quantitatively evaluate the comprehensive energy efficiency of each data block backup, providing a basis for evaluating the quality of data block backup.

[0027] As a further method, the backup quality of each data block is evaluated according to the comprehensive energy efficiency evaluation index of each data block backup. The specific analysis process is: extracting the data block backup comprehensive energy efficiency evaluation index threshold from the data processing system database, comparing each data block backup comprehensive energy efficiency evaluation index with the data block backup comprehensive energy efficiency evaluation index threshold, if a data block backup comprehensive energy efficiency evaluation index is higher than or equal to the data block backup comprehensive energy efficiency evaluation index threshold, then the data block backup quality is evaluated as qualified, if a data block backup comprehensive energy efficiency evaluation index is lower than the data block backup comprehensive energy efficiency evaluation index threshold, then the data block backup quality is evaluated as unqualified.

[0028] A second aspect of the present invention provides a distributed file data processing system based on cloud computing, comprising: a node screening module, a data backup module and a backup quality assessment module.

[0029] The node screening module is used to obtain the basic data of each distributed storage node, process and obtain the comprehensive performance evaluation value of each distributed storage node, and screen the nodes according to the comprehensive performance evaluation value of each distributed storage node to obtain each target node.

[0030] The data backup module is used to write data and divide the data into blocks, mark them into data blocks, and copy each data block to each target node.

[0031] The backup quality assessment module is used to collect system backup data, and obtain the backup efficiency assessment value of each data block after processing. According to the comprehensive performance evaluation value of each target node and the backup efficiency assessment value of each data block, a comprehensive analysis is performed to obtain the comprehensive energy efficiency evaluation index of each data block backup, and the backup quality of each data block is evaluated according to the comprehensive energy efficiency evaluation index of each data block backup, and the evaluation results are recorded.

[0032] A third aspect of the present invention provides a storage medium for a distributed file data processing method based on cloud computing, comprising: a data receiver, a processor, a memory and a display.

[0033] The data receiver is used to receive basic data, system write data and system backup data of each distributed storage node.

[0034] The processor is used to evaluate the comprehensive performance of each distributed storage node according to the data received by the receiver and to screen the nodes.

[0035] It is also used to evaluate the backup quality of each data block.

[0036] The memory is used to divide the system write data into blocks, mark them into data blocks, and copy each data block to each target node.

[0037] The display is used to receive and display the backup quality assessment results of each data block of the processor.

[0038] Compared with the prior art, the embodiments of the present invention have at least the following advantages or beneficial effects:

[0039] (1) The present invention provides a distributed file data processing method, system and storage medium based on cloud computing, which continuously monitors and analyzes the comprehensive performance of data block storage nodes and the backup efficiency of data blocks. By ensuring that the node performance matches the backup efficiency, it can not only help achieve more efficient resource allocation, but also improve the reliability and business continuity of the entire backup system, reduce backup failures caused by hardware failures, thereby improving backup quality and ensuring the efficient operation of the backup system.

[0040] (2) The present invention helps to comprehensively evaluate the comprehensive performance of nodes from multiple perspectives by comprehensively analyzing the throughput, read and write speed, network latency, network bandwidth, normal operation time and resource utilization of nodes. Multi-dimensional data provides more accurate performance evaluation and reduces misjudgment caused by single indicator deviation. This provides strong support for system optimization, fault prevention, resource management and business decision-making.

[0041] (3) The present invention provides a multi-dimensional perspective for evaluating backup efficiency by comprehensively analyzing the total backup time, average transmission rate, backup success rate and fault recovery time of each data block, which helps to ensure the optimization and reliability of data backup operations, thereby providing strong support for data protection and business continuity. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] The present invention is further described using the accompanying drawings, but the embodiments in the accompanying drawings do not constitute any limitation to the present invention. A person skilled in the art can obtain other drawings based on the following drawings without creative work.

[0043] Figure 1 It is a schematic diagram of the method flow of the present invention.

[0044] Figure 2 It is a schematic diagram of system module connection of the present invention.

[0045] Figure 3 It is a schematic diagram of the functional relationship between the data block backup efficiency evaluation value and the data block backup comprehensive energy efficiency evaluation index involved in an embodiment of the present invention. DETAILED DESCRIPTION

[0046] The technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0047] Reference Figure 1 As shown, the first aspect of the present invention provides a distributed file data processing method based on cloud computing, including: S1. Acquire basic data of each distributed storage node, process to obtain a comprehensive performance evaluation value of each distributed storage node, and screen the nodes according to the comprehensive performance evaluation value of each distributed storage node to obtain each target node.

[0048] Specifically, the basic data of each distributed storage node includes: node storage performance data, node network performance data, node average uptime and node resource utilization.

[0049] The node stores performance data, including throughput and read and write speeds at each monitoring point in time.

[0050] The node network performance data includes network delay and network bandwidth at each time monitoring point.

[0051] It should be explained that the average uptime of the node in this embodiment refers to the average time that the node remains available. It reflects the reliability and stability of the node. The longer the average uptime of the node, the higher the reliability of the system. The throughput of the node refers to the ability of the node to process data per unit time, usually measured in the number of read and write operations per second. The read and write speed of the node refers to the data transmission rate when the node performs read and write operations, usually in megabytes per second (MB / s). The network latency of the node refers to the time required for data transmission between nodes, usually in milliseconds (ms). The network bandwidth of the node refers to the data transmission rate of the node network interface, usually in megabits per second (Mbps). Node resource utilization refers to the ratio of the resources actually used by the node during the monitoring period to its total resources, and the resources include CPU, memory and disk space. The node resource utilization is equal to the average value of CPU utilization, memory utilization and disk space utilization.

[0052] It should be explained that the throughput and read and write speed of the node can be obtained by benchmarking through performance testing tools (such as IOMeter, FIO). Network latency can be measured using network diagnostic tools (such as ping, traceroute) or professional network performance monitoring tools (such as Wireshark, SolarWinds). The network bandwidth data of the node can be obtained through network bandwidth testing services (such as Speedtest.net) or network monitoring tools. The operating status of the node can be tracked through system monitoring tools (such as Nagios, Zabbix), and the normal operation time can be calculated based on the normal operation start time and end time. Resource utilization data can be obtained through the built-in monitoring commands of the operating system (such as top, vmstat, iostat) or professional system monitoring tools.

[0053] Furthermore, the comprehensive performance evaluation value of each distributed storage node is processed and obtained. The specific process is: extracting the critical node normal operation time, critical node resource utilization, critical node throughput, critical node read and write speed, critical node network delay and critical node network bandwidth from the data processing system database.

[0054] Based on the basic data of each distributed storage node, a comprehensive analysis is performed to obtain a comprehensive performance evaluation value of each distributed storage node. The comprehensive performance evaluation value of each distributed storage node is used to quantitatively evaluate the reliability of each distributed storage node in performing data backup, providing a basis for node screening.

[0055] In a specific embodiment, the numerical expression of the comprehensive performance evaluation value of each distributed storage node is:

[0056]

[0057] In the formula, ZH r represents the comprehensive performance evaluation value of the rth distributed storage node, r represents the number of each distributed storage node, r = 1, 2, 3, ..., h, h represents the total number of distributed storage nodes, i represents the number of each time monitoring point, i = 1, 2, 3, ..., t, t represents the total number of time monitoring points, YT r represents the average uptime of the rth distributed storage node, YL r represents the resource utilization of the rth distributed storage node, TL ri represents the throughput of the rth distributed storage node at the i-th time monitoring point, DX ri WY represents the read and write speed of the rth distributed storage node at the i-th time monitoring point, ri represents the network delay of the rth distributed storage node at the i-th time monitoring point, WK rirepresents the network bandwidth of the r-th node at the i-th time monitoring point, ΔYT represents the normal operation time of the critical node, ΔYL represents the resource utilization of the critical node, ΔTL represents the throughput of the critical node, ΔDX represents the read and write speed of the critical node, ΔWY represents the network delay of the critical node, ΔWK represents the network bandwidth of the critical node, σ1 represents the node comprehensive performance impact factor corresponding to the preset normal operation time, σ2 represents the node comprehensive performance impact factor corresponding to the preset resource utilization, σ3 represents the node comprehensive performance impact factor corresponding to the preset throughput, σ4 represents the node comprehensive performance impact factor corresponding to the preset read and write speed, σ5 represents the node comprehensive performance impact factor corresponding to the preset network delay, and σ6 represents the node comprehensive performance impact factor corresponding to the preset network bandwidth.

[0058] It needs to be explained that the greater the node's uptime, throughput, read / write speed and network bandwidth, and the smaller the node's resource utilization and network latency during uptime, the greater the corresponding comprehensive performance evaluation value of the distributed storage node, indicating that the distributed storage node is more suitable for data backup.

[0059] It should be explained that in this embodiment, σ1 represents the node comprehensive performance impact factor corresponding to the preset normal operation time, σ2 represents the node comprehensive performance impact factor corresponding to the preset resource utilization rate, σ3 represents the node comprehensive performance impact factor corresponding to the preset throughput, σ4 represents the node comprehensive performance impact factor corresponding to the preset read and write speed, σ5 represents the node comprehensive performance impact factor corresponding to the preset network delay, σ6 represents the node comprehensive performance impact factor corresponding to the preset network bandwidth, and the values ​​of these impact factors respectively represent the degree of influence of the node's normal operation time, resource utilization rate, throughput, read and write speed, network delay and network bandwidth unit values ​​on the node comprehensive performance. When used, these impact factors can be directly obtained from the data processing system database. The values ​​of these impact factors are preset in the data processing system database. For example, the node's normal operation time, resource utilization rate, throughput, read and write speed, network delay and network bandwidth are respectively mapped with the impact factors preset in the data processing system database. The real-time node's normal operation time, resource utilization rate, throughput, read and write speed, network delay and network bandwidth are input into the mapping set to obtain the corresponding node comprehensive performance impact factor. These mapping relationships can be one-to-one or many-to-one. The value range of all impact factors is between 0 and 1, representing the degree from no impact to maximum impact.

[0060] It needs to be explained that storage throughput is limited by network bandwidth. If the network bandwidth is insufficient, even if the storage device has a fast read and write speed, the bottleneck of data transmission will limit the overall throughput. The read and write speed of storage may be affected by network latency. In a high-latency network environment, even if the storage device has good performance, the read and write operations of data will be slow. Storage operations may increase the use of CPU and memory. If the resource utilization rate is too high, it may reduce storage performance. The stability of network performance directly affects the uptime of the node. Network interruptions or high latency may cause node service unavailability, thereby reducing uptime. If the resource utilization of the node is high for a long time, it may cause system overload, which in turn causes failures and reduces uptime. Reasonable allocation of resources to ensure that no node is over-utilized or under-utilized can help improve system stability and uptime.

[0061] It should be explained that by analyzing the throughput, read and write speed, network latency, network bandwidth, uptime, and resource utilization of the nodes, it is easier to identify performance bottlenecks, thereby helping to optimize in a targeted manner and improve the overall performance of the system. Continuous monitoring of these parameters can also help predict potential performance problems, such as resource overload or increased network latency, so that measures can be taken in advance. It is beneficial to improve the stability and reliability of the system and reduce the occurrence of failures. Understanding the resource utilization of the nodes can help allocate resources more efficiently and avoid resource waste. Network latency and storage read and write speeds directly affect the response time of the service, and optimizing these parameters can improve the user experience. The length of uptime is directly related to the availability of the service, and comprehensive analysis helps ensure the high availability of the nodes.

[0062] It should be explained that this embodiment helps to comprehensively evaluate the comprehensive performance of nodes from multiple perspectives by comprehensively analyzing the throughput, read and write speed, network latency, network bandwidth, normal operation time and resource utilization of nodes. Multi-dimensional data provides more accurate performance evaluation and reduces misjudgment caused by deviation of a single indicator. This provides strong support for system optimization, fault prevention, resource management and business decision-making.

[0063] Furthermore, the nodes are screened to obtain target nodes. The specific screening process is: extracting the comprehensive performance evaluation threshold of the distributed storage nodes from the data processing system database, comparing the comprehensive performance evaluation value of each distributed storage node with the comprehensive performance evaluation threshold of the distributed storage node; if the comprehensive performance evaluation value of a distributed storage node is higher than or equal to the comprehensive performance evaluation threshold of the distributed storage node, the performance evaluation of the distributed storage node is considered qualified and marked as a target node; if the comprehensive performance evaluation value of a distributed storage node is lower than the comprehensive performance evaluation threshold of the distributed storage node, the performance evaluation of the distributed storage node is considered unqualified.

[0064] Count the target nodes.

[0065] S2. The file processing system writes the data and divides the data into blocks, marks them into data blocks, and copies each data block to each target node.

[0066] Specifically, each data block is copied to each target node. The specific process is as follows: U1. The file processing system receives a data write request from a user and temporarily stores the data in a buffer of the system.

[0067] U2. Cut the temporarily stored data into pieces according to the preset block size, generate data blocks, and mark them as data blocks.

[0068] It should be added that the block size can be 4MB or 16MB.

[0069] It should be added that a unique identifier is assigned to each data block for easy management and tracking.

[0070] U3. Match the target node for each data block according to the system's load balancing algorithm and establish a network connection with the target node.

[0071] U4. Transmit each data block in parallel to the corresponding target node, and mark the node storing the data block as each storage node for each data block.

[0072] In a specific embodiment, there is a one-to-many relationship between data blocks and storage nodes, and the data of a data block needs to be backed up in multiple storage nodes.

[0073] S3. Collect system backup data and obtain the backup efficiency evaluation value of each data block after processing.

[0074] Specifically, the backup efficiency evaluation value of each data block is obtained through processing, and the specific process is as follows: the system backs up data, including the total backup time, average transmission rate, backup success rate and fault recovery time of each data block.

[0075] It should be explained that the total backup time of the data block refers to the total time required to back up the data block to each storage node. The average transmission rate refers to the average rate at which the data block is transmitted to each storage node. The backup success rate refers to the ratio of the number of storage nodes that successfully back up the data block to the total number of storage nodes that need to back up the data block when the data block is backed up. The fault recovery time refers to the time required to restore the backup if a fault occurs during the backup process.

[0076] It should be explained that system monitoring tools (such as Nagios, Zabbix, Prometheus, etc.) can be used to track backup tasks and record duration. Network monitoring tools (such as Wireshark, NetFlow, etc.) can be used to monitor network traffic during backup and calculate the average rate. The backup software will provide a status report after the operation is completed, indicating whether the backup was successful. Fault management systems (such as ServiceNow, JIRA, etc.) can be used to track fault reports and recovery times.

[0077] The total backup time of critical data blocks, the average transmission rate of critical data blocks, the backup success rate of critical data blocks and the failure recovery time of critical data blocks are extracted from the data processing system database.

[0078] According to the system backup data, the backup efficiency evaluation value of each data block is processed to obtain the backup efficiency evaluation value of each data block, which is used to quantitatively evaluate the backup efficiency of each data block and provide a basis for quality evaluation of the data block.

[0079] In a specific embodiment, the numerical expression of the backup efficiency evaluation value of each data block is:

[0080]

[0081] In the formula, BX n represents the backup efficiency evaluation value of the nth data block, e represents a natural constant, n represents the number of each data block, n=1,2,3,...,m, m represents the total number of data blocks, ZB n Indicates the total backup time of the nth data block, PS n Indicates the average transmission rate of the nth data block, BC n Indicates the backup success rate of the nth data block, HF n represents the fault recovery time of the nth data block, ΔZB represents the total backup time of the critical data block, ΔPS represents the average transmission rate of the critical data block, ΔBC represents the backup success rate of the critical data block, ΔHF represents the fault recovery time of the critical data block, θ1 represents the data block backup efficiency correction factor corresponding to the preset total backup time, θ2 represents the data block backup efficiency correction factor corresponding to the preset average transmission rate, θ3 represents the data block backup efficiency correction factor corresponding to the preset backup success rate, and θ4 represents the data block backup efficiency correction factor corresponding to the preset fault recovery time.

[0082] It should be explained that, when the average transmission rate and backup success rate of the data block are greater, and the total backup time and fault recovery time of the data block are smaller, the corresponding data block backup efficiency evaluation value is greater, indicating that the backup efficiency of the data block is higher.

[0083] It should be explained that, in this embodiment, θ1 represents the data block backup efficiency correction factor corresponding to the preset total backup time, θ2 represents the data block backup efficiency correction factor corresponding to the preset average transmission rate, θ3 represents the data block backup efficiency correction factor corresponding to the preset backup success rate, and θ4 represents the data block backup efficiency correction factor corresponding to the preset fault recovery time. The values ​​of these correction factors respectively represent the degree of influence of the total backup time, average transmission rate, backup success rate and fault recovery time unit values ​​of the data block on the data block backup efficiency. When used, these correction factors can be directly obtained from the data processing system database. The values ​​of these correction factors are preset in the data processing system database. For example, the total backup time, average transmission rate, backup success rate and fault recovery time of the data block are respectively mapped with the correction factors preset in the data processing system database to form a mapping set. The total backup time, average transmission rate, backup success rate and fault recovery time of the real-time data block are input into the mapping set to obtain the corresponding data block backup efficiency correction factor. These mapping relationships can be one-to-one or many-to-one. The value range of all correction factors is between 0 and 1, indicating the degree from no influence to maximum influence.

[0084] It needs to be explained that the total backup time is usually the result of dividing the data block size by the average transfer rate. If the average transfer rate increases, the total backup time usually decreases, and vice versa. If the backup success rate is low, the backup may need to be retried, which will increase the total backup time. If the average transfer rate drops due to network or storage node performance issues, the risk of backup failure may increase, thereby reducing the backup success rate. If the fault recovery time is short, it means that the system can recover from the failure quickly, which helps to maintain or improve the backup success rate. Conversely, a long fault recovery will reduce the backup success rate because it may cause the backup task to timeout or fail.

[0085] It should be explained that this embodiment provides a multi-dimensional perspective for evaluating backup efficiency by comprehensively analyzing the total backup time, average transmission rate, backup success rate and fault recovery time of each data block, which helps to ensure the optimization and reliability of data backup operations, thereby providing strong support for data protection and business continuity. By comprehensively analyzing these parameters, network and storage resources can also be used more effectively to avoid resource waste. It can also help predict future backup needs, plan resource expansion, and ensure that the backup system can remain efficient as the amount of data grows. It can also be used to build a monitoring system and alarm mechanism to promptly notify the administrator when the backup efficiency decreases or a failure occurs. It can also help diagnose the problem, such as whether it is due to a network problem, a storage device failure, or other system problems.

[0086] S4. According to the comprehensive performance evaluation value of each target node and the backup efficiency evaluation value of each data block, a comprehensive analysis is performed to obtain the comprehensive energy efficiency evaluation index of each data block backup, and the backup quality of each data block is evaluated according to the comprehensive energy efficiency evaluation index of each data block backup, and the evaluation results are recorded.

[0087] Specifically, a comprehensive analysis is performed to obtain a comprehensive energy efficiency evaluation index for backup of each data block. The specific analysis process is: averaging the comprehensive performance evaluation values ​​of each storage node corresponding to each data block to obtain a comprehensive performance evaluation value of each data block storage node.

[0088] In a specific embodiment, if one data block corresponds to three storage nodes, the numerical expression of the comprehensive performance evaluation value of the data block storage node is: Among them, ZH n Indicates the comprehensive performance evaluation value of the nth data block storage node, ZH n-1 Indicates the comprehensive performance evaluation value of the first storage node corresponding to the nth data block, ZH n-2 Indicates the comprehensive performance evaluation value of the second storage node corresponding to the nth data block, ZH n-3 Represents the comprehensive performance evaluation value of the third storage node corresponding to the nth data block.

[0089] According to the comprehensive performance evaluation value of each data block storage node and the evaluation value of each data block backup efficiency, a comprehensive analysis is performed to obtain a comprehensive energy efficiency evaluation index for each data block backup. The comprehensive energy efficiency evaluation index for each data block backup is used to quantitatively evaluate the comprehensive energy efficiency of each data block backup, providing a basis for evaluating the quality of data block backup.

[0090] In a specific embodiment, the numerical expression of the comprehensive energy efficiency evaluation index of each data block backup is:

[0091]

[0092] In the formula, KN n represents the comprehensive energy efficiency evaluation index of the nth data block backup, JX n Indicates the comprehensive performance evaluation value of the nth data block storage node, BX n Indicates the backup efficiency evaluation value of the nth data block. The comprehensive energy efficiency impact characteristic factor of the data block backup corresponding to the comprehensive performance evaluation value of the preset data block storage node, Indicates the data block backup comprehensive energy efficiency impact characteristic factor corresponding to the preset data block backup efficiency evaluation value.

[0093] It should be explained that when the comprehensive performance evaluation value of the data block storage node and the data block backup efficiency evaluation value are greater, the corresponding data block backup comprehensive energy efficiency evaluation index is greater, indicating that the comprehensive energy efficiency of the data block backup is higher and the backup quality of the data block is better.

[0094] It should be explained that in this embodiment The comprehensive energy efficiency impact characteristic factor of the data block backup corresponding to the comprehensive performance evaluation value of the preset data block storage node, The data block backup comprehensive energy efficiency impact characteristic factors corresponding to the preset data block backup efficiency evaluation values ​​are represented. The values ​​of these impact characteristic factors respectively represent the degree of influence of the comprehensive performance evaluation value of the data block storage node and the unit value of the data block backup efficiency evaluation value on the comprehensive energy efficiency of the data block backup. When used, these impact characteristic factors can be directly obtained from the data processing system database. The values ​​of these impact characteristic factors are preset in the data processing system database. For example, the comprehensive performance evaluation value of the data block storage node and the data block backup efficiency evaluation value respectively form a mapping set with the impact characteristic factors preset in the data processing system database. The comprehensive performance evaluation value and the data block backup efficiency evaluation value of the real-time data block storage node are input into the mapping set to obtain the corresponding data block backup comprehensive energy efficiency impact characteristic factors. These mapping relationships can be one-to-one or many-to-one. The value range of all impact characteristic factors is between 0 and 1, indicating the degree from no impact to maximum impact.

[0095] like Figure 3 As shown, curve a represents the relationship between the corresponding data block backup efficiency evaluation value and the data block backup comprehensive energy efficiency evaluation index when the comprehensive performance evaluation value of the data block storage node is 1, curve b represents the relationship between the corresponding data block backup efficiency evaluation value and the data block backup comprehensive energy efficiency evaluation index when the comprehensive performance evaluation value of the data block storage node is 2, and curve c represents the relationship between the corresponding data block backup efficiency evaluation value and the data block backup comprehensive energy efficiency evaluation index when the comprehensive performance evaluation value of the data block storage node is 3.

[0096] It should be added that, in this embodiment, the data block backup comprehensive energy efficiency impact characteristic factor corresponding to the comprehensive performance evaluation value of the data block storage node is set to 0.5, and the data block backup comprehensive energy efficiency impact characteristic factor corresponding to the data block backup efficiency evaluation value is set to 0.8.

[0097] It should be explained that when the node has good performance and fast processing speed, the backup time can be shortened. When the node has high network bandwidth, the data block transmission rate can be increased. At the same time, stable node performance can improve the backup success rate and reduce backup failures caused by hardware failures. High-performance nodes may also have faster fault detection and recovery capabilities. Evaluating the comprehensive performance of the node can provide expected indicators for backup efficiency, and the actual backup efficiency evaluation value can verify whether the comprehensive performance of the node meets expectations. If the backup efficiency is lower than expected, the backup strategy can be adjusted or the node hardware can be upgraded based on the comprehensive performance evaluation value of the node.

[0098] It should be explained that in this embodiment, by continuously monitoring and evaluating the comprehensive performance of the data block storage nodes and the backup efficiency of the data blocks, the optimization and efficient operation of the backup system can be ensured. By ensuring that the nodes have sufficient comprehensive performance to handle the backup tasks, errors and data corruption in the backup process can be reduced, thereby improving the backup quality. It can also help identify which nodes need more resources and which nodes may have excess resources, thereby achieving more efficient resource allocation. By ensuring that the node performance matches the backup efficiency, the reliability of the entire backup system can be improved and backup failures caused by hardware failures can be reduced. If it is found that the backup efficiency of some data blocks is lower than that of other data blocks, the backup strategy can be adjusted, such as changing the backup time, optimizing the data block size, or selecting a more appropriate backup method. High-quality backup is a key component of business continuity. By ensuring the quality of backup, data can be restored quickly and business interruption time can be reduced.

[0099] Furthermore, the backup quality of each data block is evaluated according to the comprehensive energy efficiency evaluation index of each data block backup. The specific analysis process is: extracting the data block backup comprehensive energy efficiency evaluation index threshold from the data processing system database, comparing each data block backup comprehensive energy efficiency evaluation index with the data block backup comprehensive energy efficiency evaluation index threshold, if a data block backup comprehensive energy efficiency evaluation index is higher than or equal to the data block backup comprehensive energy efficiency evaluation index threshold, then the data block backup quality is evaluated as qualified; if a data block backup comprehensive energy efficiency evaluation index is lower than the data block backup comprehensive energy efficiency evaluation index threshold, then the data block backup quality is evaluated as unqualified.

[0100] In a specific embodiment, the number of data blocks whose quality is assessed as unqualified is recorded, and an unqualified data block index parameter is extracted from the data processing system database. If the number of unqualified data blocks is higher than the unqualified data block index parameter, a risk warning is required for the data processing system.

[0101] Specifically, the process of risk warning is as follows: V1. Once the risk assessment results meet the warning conditions, the warning mechanism is immediately activated. Warning information containing risk level, possible impact, recommended measures, etc. is produced. V2. Warning information is sent to the maintenance team and relevant persons in charge of the data processing system. V3. The maintenance team activates the emergency plan and takes corresponding measures based on the warning information. The system is checked to find out the reasons for the increase in unqualified data blocks. V4. Solve the problems found in a targeted manner, such as optimizing the data processing process, repairing system vulnerabilities, etc. After the problem is solved, continue to monitor the situation of unqualified data blocks to ensure that the problem is effectively solved. V5. After taking measures, re-evaluate the situation of unqualified data blocks. If the number of unqualified data blocks drops below the indicator parameter and the system runs stably for a period of time, the warning can be lifted. V6. Summarize the risk warning process and form a report. Based on the summary report, propose system improvement and preventive measures to avoid similar risks from happening again.

[0102] Reference Figure 2 As shown, the second aspect of the present invention provides a distributed file data processing system based on cloud computing, including: a node screening module, a data backup module and a backup quality assessment module.

[0103] The node screening module is used to obtain the basic data of each distributed storage node, process and obtain the comprehensive performance evaluation value of each distributed storage node, and screen the nodes according to the comprehensive performance evaluation value of each distributed storage node to obtain each target node.

[0104] The data backup module is used to write data and divide the data into blocks, mark them into data blocks, and copy each data block to each target node.

[0105] The backup quality assessment module is used to collect system backup data, and obtain the backup efficiency assessment value of each data block after processing. According to the comprehensive performance evaluation value of each target node and the backup efficiency assessment value of each data block, a comprehensive analysis is performed to obtain the comprehensive energy efficiency evaluation index of each data block backup, and the backup quality of each data block is evaluated according to the comprehensive energy efficiency evaluation index of each data block backup, and the evaluation results are recorded.

[0106] In a specific embodiment, the data processing system database is used to store relevant data in the process of evaluating the quality of data block backup, including critical node normal operation time, critical node resource utilization, critical node throughput, critical node read and write speed, critical node network delay and critical node network bandwidth, data block backup comprehensive energy efficiency evaluation index threshold and the data extracted from the database in the above embodiment. The configuration information of the node, including hardware specifications, network configuration and other related data, can be extracted from the configuration management database. Data can also be extracted from external databases using data analysis tools (such as SQL query, data mining software, etc.) for further analysis and evaluation. Relevant data can also be obtained from third-party services or internal systems through APIs and data interfaces, such as the monitoring API of cloud service providers.

[0107] A third aspect of the present invention provides a storage medium for a distributed file data processing method based on cloud computing, comprising: a data receiver, a processor, a memory and a display.

[0108] The data receiver is used to receive basic data, system write data and system backup data of each distributed storage node.

[0109] The processor is used to evaluate the comprehensive performance of each distributed storage node according to the data received by the receiver and to screen the nodes.

[0110] It is also used to evaluate the backup quality of each data block.

[0111] The memory is used to divide the system write data into blocks, mark them into data blocks, and copy each data block to each target node.

[0112] The display is used to receive and display the backup quality assessment results of each data block of the processor.

[0113] The above contents are merely examples and explanations of the structure of the present invention. The technicians in this technical field may make various modifications or additions to the specific embodiments described or replace them in a similar manner. As long as they do not deviate from the structure of the invention or exceed the scope defined by the claims, they should all fall within the protection scope of the present invention.

Claims

1. A distributed file data processing method based on cloud computing, characterized in that: include: S1. Obtain the basic data of each distributed storage node, process and obtain the comprehensive performance evaluation value of each distributed storage node, and screen the nodes according to the comprehensive performance evaluation value of each distributed storage node to obtain each target node; S2. The file processing system writes the data and divides the data into blocks, marks them into data blocks, and copies each data block to each target node; S3. Collect system backup data and obtain backup efficiency evaluation values ​​of each data block after processing; S4. According to the comprehensive performance evaluation value of each target node and the backup efficiency evaluation value of each data block, a comprehensive analysis is performed to obtain the comprehensive energy efficiency evaluation index of each data block backup, and the backup quality of each data block is evaluated according to the comprehensive energy efficiency evaluation index of each data block backup, and the evaluation results are recorded.

2. The distributed file data processing method based on cloud computing according to claim 1 is characterized in that: The basic data of each distributed storage node specifically includes: node storage performance data, node network performance data, node average uptime and node resource utilization; The node stores performance data, including throughput and read and write speeds at each time monitoring point; The node network performance data includes network delay and network bandwidth at each time monitoring point.

3. The distributed file data processing method based on cloud computing according to claim 2 is characterized in that: The processing obtains the comprehensive performance evaluation value of each distributed storage node, and the specific process is as follows: Extracting critical node uptime, critical node resource utilization, critical node throughput, critical node read / write speed, critical node network latency, and critical node network bandwidth from a data processing system database; Based on the basic data of each distributed storage node, a comprehensive analysis is performed to obtain a comprehensive performance evaluation value of each distributed storage node. The comprehensive performance evaluation value of each distributed storage node is used to quantitatively evaluate the reliability of each distributed storage node in performing data backup, providing a basis for node screening.

4. The distributed file data processing method based on cloud computing according to claim 3 is characterized by: The nodes are screened to obtain target nodes. The specific screening process is as follows: Extracting the comprehensive performance evaluation threshold of the distributed storage node from the data processing system database, comparing the comprehensive performance evaluation value of each distributed storage node with the comprehensive performance evaluation threshold of the distributed storage node, if the comprehensive performance evaluation value of a distributed storage node is higher than or equal to the comprehensive performance evaluation threshold of the distributed storage node, then the performance evaluation of the distributed storage node is qualified and marked as a target node, if the comprehensive performance evaluation value of a distributed storage node is lower than the comprehensive performance evaluation threshold of the distributed storage node, then the performance evaluation of the distributed storage node is unqualified; Count the target nodes.

5. The distributed file data processing method based on cloud computing according to claim 1 is characterized by: The specific process of copying each data block to each target node is as follows: U1. The file processing system receives data write requests from users and temporarily stores the data in the system's buffer; U2. Cut the temporarily stored data into blocks according to the preset block size, generate data blocks, and mark them as data blocks; U3. Match the target node for each data block according to the system's load balancing algorithm and establish a network connection with the target node; U4. Transmit each data block in parallel to the corresponding target node, and mark the node storing the data block as each storage node for each data block.

6. The distributed file data processing method based on cloud computing according to claim 1 is characterized by: The backup efficiency evaluation value of each data block is obtained through processing, and the specific process is as follows: The system backup data includes the total backup time, average transmission rate, backup success rate and fault recovery time of each data block; Extracting from the data processing system database the total backup time of critical data blocks, the average transmission rate of critical data blocks, the backup success rate of critical data blocks and the failure recovery time of critical data blocks; According to the system backup data, the backup efficiency evaluation value of each data block is processed to obtain the backup efficiency evaluation value of each data block, which is used to quantitatively evaluate the backup efficiency of each data block and provide a basis for quality evaluation of the data block.

7. The distributed file data processing method based on cloud computing according to claim 6 is characterized by: The comprehensive analysis obtains the comprehensive energy efficiency evaluation index of each data block backup, and the specific analysis process is as follows: The comprehensive performance evaluation values ​​of the storage nodes corresponding to each data block are averaged to obtain the comprehensive performance evaluation value of the storage node of each data block; According to the comprehensive performance evaluation value of each data block storage node and the evaluation value of each data block backup efficiency, a comprehensive analysis is performed to obtain a comprehensive energy efficiency evaluation index for each data block backup. The comprehensive energy efficiency evaluation index for each data block backup is used to quantitatively evaluate the comprehensive energy efficiency of each data block backup, providing a basis for evaluating the quality of data block backup.

8. The distributed file data processing method based on cloud computing according to claim 7 is characterized in that: The backup quality of each data block is evaluated according to the comprehensive energy efficiency evaluation index of each data block backup. The specific analysis process is as follows: The data block backup comprehensive energy efficiency evaluation index threshold is extracted from the data processing system database, and each data block backup comprehensive energy efficiency evaluation index is compared with the data block backup comprehensive energy efficiency evaluation index threshold. If a data block backup comprehensive energy efficiency evaluation index is higher than or equal to the data block backup comprehensive energy efficiency evaluation index threshold, the data block backup quality is evaluated as qualified; if a data block backup comprehensive energy efficiency evaluation index is lower than the data block backup comprehensive energy efficiency evaluation index threshold, the data block backup quality is evaluated as unqualified.

9. A system using the distributed file data processing method based on cloud computing as claimed in any one of claims 1 to 8, characterized in that: include: Node screening module, data backup module and backup quality assessment module; The node screening module is used to obtain the basic data of each distributed storage node, process and obtain the comprehensive performance evaluation value of each distributed storage node, and screen the nodes according to the comprehensive performance evaluation value of each distributed storage node to obtain each target node; The data backup module is used to write data and divide the data into blocks, mark them into data blocks, and copy each data block to each target node; The backup quality assessment module is used to collect system backup data, and obtain the backup efficiency assessment value of each data block after processing. According to the comprehensive performance evaluation value of each target node and the backup efficiency assessment value of each data block, a comprehensive analysis is performed to obtain the comprehensive energy efficiency evaluation index of each data block backup, and the backup quality of each data block is evaluated according to the comprehensive energy efficiency evaluation index of each data block backup, and the evaluation results are recorded.

10. A storage medium using the distributed file data processing method based on cloud computing as claimed in any one of claims 1 to 8, characterized in that: include: data receiver, processor, memory and display; The data receiver is used to receive basic data, system write data and system backup data of each distributed storage node; The processor is used to evaluate the comprehensive performance of each distributed storage node and screen the nodes according to the data received by the receiver; It is also used to evaluate the backup quality of each data block; The memory is used to divide the system write data into blocks, mark them into data blocks, and copy each data block to each target node; The display is used to receive and display the backup quality assessment results of each data block of the processor.

Images