A distributed file processing method and system

Abstract

The present invention relates to the technical field of distributed file processing, specifically a distributed file processing method and system, comprising the following steps: based on task allocation nodes and data exchange interfaces, extracting the path characteristics of task input and output files, analyzing the file exchange order of the task path, and calculating the topological structure and length of the path. In the present invention, by combining task status monitoring, real-time adjustment of dependent path weights and task priority tags, the flexibility and adaptability of task allocation are achieved, the request frequency and time period records are used to optimize resource allocation and short-term task sequence, effectively alleviate high-frequency request resource conflicts, and improve resource allocation efficiency. Through the analysis of node historical resource distribution data and time distribution trends, the task distribution is adjusted in combination with differences, resource loads are balanced, and the stability and balance of the system are enhanced. The file transmission path is optimized based on the load status and task execution path, and the concurrent task distribution strategy is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of distributed file processing, and in particular to a distributed file processing method and system. Background Art

[0002] The field of distributed file processing technology includes distributed systems and related technologies for large-scale data storage and processing, mainly involving distributed storage, management and access to files in multi-node systems. The core content of this technology field is to achieve efficient data processing and storage through the collaboration of multiple computing nodes, thereby maximizing the utilization of system resources. In distributed file processing technology, common research directions include block storage and retrieval of files, data synchronization between nodes, fault-tolerant mechanisms, and efficient distributed index structures. This field usually uses peer-to-peer networks, master-slave architectures, or hybrid architectures to manage data distribution, and achieves performance improvements by optimizing communication protocols and node scheduling mechanisms.

[0003] Among them, the distributed file processing method refers to completing the storage, distribution and processing of files in a multi-node environment by designing specific file processing logic and distribution strategies. The subject of this patent is aimed at the problem of how to reasonably allocate file storage locations and efficiently complete file access in a multi-node system. Specifically, it is based on data block and copy mechanisms, and by establishing a logical mapping relationship between storage nodes and designing a distributed file directory, the file location can be quickly located. Task allocation and scheduling strategies are usually used in the file processing process to ensure that the multi-node system works together to complete the storage, reading and modification operations of files.

[0004] The existing technology lacks accurate identification of dynamic changes in paths and critical paths, resulting in rough task priority division and affecting task scheduling efficiency. The weights of task-dependent paths are not updated in a timely manner, and the static adjustment method is prone to lead to priority imbalance in emergencies, prolonging task processing time. Resource allocation does not comprehensively consider request frequency and time period records, and is only allocated based on a single load parameter, which can easily cause resource conflicts under high-frequency requests and reduce allocation effects. The analysis depth of historical resource distribution data is insufficient, and the time trend is not used to adjust the task order, resulting in the difficulty in effectively improving the uneven distribution of resources. File transfer path optimization lacks analysis of the correlation between load status and tasks, which affects the distribution efficiency of multi-node concurrent tasks and increases system operation pressure. Summary of the invention

[0005] The purpose of the present invention is to solve the shortcomings in the prior art and to propose a distributed file processing method and system.

[0006] In order to achieve the above object, the present invention adopts the following technical solution: a distributed file processing method, comprising the following steps:

[0007] S1: Based on the task allocation node and data exchange interface, extract the path characteristics of the task input and output files, analyze the exchange order of the file storage path, calculate the topological structure and length of the path, filter the key path according to the path length and mark the task priority, and generate the task priority sequence value in order;

[0008] S2: Based on the task priority sequence value and in combination with the task status monitoring data of the resource manager, analyze the file dependency path changes after the task is completed, recalculate the dependency path weights, update the task priority mark according to the weight changes, adjust the association between the path and the node, and generate a dynamic priority adjustment value;

[0009] S3: Based on the dynamic priority adjustment value, filter the storage nodes according to the request frequency, extract the time period records of the conflicting resources, compare the resource request frequencies and mark the conflicting states, allocate priority resources to the conflicting tasks, adjust the execution order of the short-term tasks, and generate the short-term resource allocation results;

[0010] S4: Based on the short-term resource allocation result and node load parameters, extract historical storage distribution data, analyze the time distribution trend of resource usage, calculate the difference between resource status and current distribution, adjust the task distribution order according to the conflict trend, and generate a conflict risk scheduling order value;

[0011] S5: Based on the conflict risk scheduling order value, combined with the file transfer request and the load status of the resource manager, extract high-load storage nodes and tasks, analyze the execution path of the node tasks, adjust the relationship between the file transfer path and the node tasks, optimize the resource distribution order of concurrent tasks, and calculate the optimized task distribution rate.

[0012] The task priority sequence value includes the topological structure of the task path, the path length, the critical path mark, and the task priority mark; the dynamic priority adjustment value specifically includes the dependent path weight, the task priority mark adjustment value, and the path-node association relationship; the short-term resource allocation result includes the conflict resource time period record, the resource request frequency, the conflict status mark, and the short-term task execution order; the conflict risk scheduling sequence value includes the node load parameters, the resource time distribution trend, the resource distribution difference, and the conflict trend adjustment result.

[0013] As a further solution of the present invention, the step of obtaining the task priority sequence value is specifically as follows:

[0014] S101: extracting input and output file path features of multiple nodes in the task path, calculating the path topology structure according to the uniqueness and hierarchical relationship of the path, marking the topology structure in layers according to the relationship between the path nodes, counting the connection relationship of each layer of path nodes and the number of associated file exchanges, and generating hierarchical features of the path;

[0015] S102: performing file exchange sequence analysis on the hierarchical features of the path, sorting and counting the exchange sequence according to the time sequence of file exchange between nodes, counting the path coverage and the uniformity of the coverage based on the arrangement results of the file exchange sequences of multiple paths, calculating the length feature of the path, and generating the path length feature;

[0016] S103: Screen the path length features, remove the paths with lengths shorter than the global average length by combining the length feature values ​​of multiple paths, and calculate the comprehensive weighted lengths of the remaining paths using the formula:

[0017] ;

[0018] The path with the largest weight length value among the screening paths is marked as the critical path;

[0019] in, Represents the comprehensive weight length of the path, Represents the path The length of the segment, represents the weight of the road segment length, is the path criticality adjustment coefficient, Represents the maximum length of multiple segments in the path, Represents the total number of segments in the path, is the node level adjustment coefficient;

[0020] S104: Assigning task priorities to the critical path, prioritizing nodes according to the hierarchical relationship of nodes in the path topology structure, adjusting the node order in combination with the timing of the file exchange sequence, and generating a task priority sequence value based on the adjusted node order.

[0021] As a further solution of the present invention, the step of obtaining the dynamic priority adjustment value is specifically as follows:

[0022] S201: extracting dependent path change information after task completion based on the task priority sequence value and task status monitoring data provided by the resource manager, performing a preliminary analysis on the dependent path through the task completion time between nodes, calculating the path weight change, and generating a preliminary dependent path weight;

[0023] S202: recalculate the preliminary dependency path weights, comprehensively consider task urgency, resource utilization efficiency and delay time, and use the formula:

[0024] ;

[0025] Generate updated dependency path weights;

[0026] in, represents the updated path weight, represents the original path weight, Represents resource utilization efficiency, Represents the task delay time, is the time after the task is completed. is the time decay constant, Indicates the number of tasks in the dependency path;

[0027] S203: According to the updated dependent path weights, count the changes in the task priority tags, generate a tag adjustment basis by comparing the resource allocation amount and task completion rate of the node, and reallocate the task priorities to generate the task priorities before and after the adjustment;

[0028] S204: adjusting the path association relationship of the task nodes in combination with the task priorities and path weight changes before and after the adjustment, reconfiguring the path order relationship of the nodes according to the time sensitivity of the tasks, and generating a dynamic priority adjustment value.

[0029] As a further solution of the present invention, the step of obtaining the short-term resource allocation result is specifically:

[0030] S301: extracting the request frequency based on the dynamic priority adjustment value, counting the number of accesses to the resource node within the request cycle, calculating the average request frequency according to the total number of requests for each resource node, screening resource nodes with high request frequencies by comparing the average request frequencies of the nodes, and generating a list of high-frequency request resource nodes;

[0031] S302: extract the time period records of conflicting resources according to the high-frequency request resource node list, count the concurrent requests for each resource in the conflicting time period, and calculate the priority allocation weight of the conflicting resources using the formula:

[0032] ;

[0033] Generate a list of conflicting resource allocation weights;

[0034] in, represents the resource allocation weight, Indicates The request intensity of the resource, Indicates the request frequency, Indicates the number of concurrent conflicts, is the conflict impact adjustment coefficient, represents the global mean of the number of conflicts, Indicates the total number of conflicting resources;

[0035] S303: adjusting the conflicting resource status mark according to the conflicting resource allocation weight list, allocating priority resources according to the weight values, reordering the priority allocated resource nodes in combination with the conflicting time periods, adjusting the execution order of the short-term tasks, and generating a priority allocation task order;

[0036] S304: recalculate the resource utilization rate of the short-term task according to the priority task allocation order, optimize the task allocation plan in combination with the current load of the resource node, adjust the task allocation logic according to the relationship between the resource utilization rate and the current load, and generate a short-term resource allocation result.

[0037] As a further solution of the present invention, the step of obtaining the conflict risk scheduling order value is specifically as follows:

[0038] S401: extracting node resource distribution data based on the short-term resource allocation result and the node load parameter, determining the load distribution difference of multiple nodes by comparing and analyzing the resource usage status of the node and the current load value, and generating a node load difference analysis result;

[0039] S402: Calculate the time distribution trend of resource usage according to the node load difference analysis result, evaluate the resource usage fluctuation of the time period by counting the resource allocation frequency and node load changes in multiple time periods, identify the key time period, and generate a resource usage time distribution trend report;

[0040] S403: Analyze the resource usage time distribution trend report, calculate the difference between the resource status and the current distribution, and quantitatively evaluate the resource distribution difference and conflict risk using the formula:

[0041] ;

[0042] Calculate the resource distribution difference and generate resource status and distribution difference evaluation results;

[0043] in, Indicates the resource distribution difference, Indicates Resource usage status for a time period, Indicates the average value of resource usage status. Indicates the standard deviation of resource usage status, Indicates the total number of time periods;

[0044] S404: Based on the resource status and distribution difference evaluation results, the distribution order of tasks is adjusted by calculating the conflict trend, the matching degree between tasks and resources is optimized, an adjusted task distribution order is generated, and a conflict risk scheduling order value is obtained.

[0045] As a further solution of the present invention, the step of obtaining the optimization task distribution rate is specifically as follows:

[0046] S501: Based on the conflict risk scheduling order value, combined with the file transfer request and the load status of the resource manager, analyzing the nodes and associated tasks with uneven resource distribution, extracting resource distribution deviation data by comparing the resource distribution and load of the nodes and tasks, and generating a resource uneven distribution report;

[0047] S502: Analyze the execution path of the node task according to the uneven resource distribution report, identify the execution efficiency bottleneck caused by uneven resource distribution by comparing and calculating the relationship between the execution timing of the node task and the resource distribution, extract task optimization requirements, and generate the node task execution path analysis result;

[0048] S503: Based on the analysis result of the node task execution path, the relationship between the file transmission path and the node task is adjusted by reconfiguring the file transmission path, using the formula:

[0049] ;

[0050] Reallocate node resources and optimize task paths, and calculate and generate the optimized task distribution rate;

[0051] in, represents the optimization task distribution rate, It is The resource usage of each task, represents the average resource usage of all tasks, represents the standard deviation of resource utilization, Represents the total number of tasks, indicating the scope of statistics and calculations on resource usage of all tasks.

[0052] A distributed file processing system, the distributed file processing system is used to execute the above-mentioned distributed file processing method, the system comprises:

[0053] The task priority analysis module extracts the file exchange order in the path based on the path characteristics of the task input and output files, calculates the path topology structure and length, marks the priority of the tasks in the path, arranges the priorities and generates the path priority sequence value;

[0054] The dynamic path adjustment module calculates the weight change of the dependent path after the task is completed based on the path priority sequence value and the task status data of the resource node, compares the path weight values ​​before and after the update, re-marks the task priority and adjusts the node association information, and establishes dynamic priority adjustment parameters;

[0055] The short-term resource scheduling module extracts the request frequency data and conflict time period records of the resource nodes based on the dynamic priority adjustment parameters, calculates the resource request frequency within the time period and marks the conflict status, allocates priority resources and adjusts the execution order of short-term tasks, calculates the resource distribution difference value of the task allocation, and generates the short-term resource distribution adjustment rate;

[0056] The conflict risk assessment module extracts the distribution data and load parameters of resource nodes based on the short-term resource distribution adjustment rate, calculates the resource time distribution trend value and compares it with the current distribution trend value, identifies resource conflict trend changes, analyzes task distribution sequence differences and adjusts node associations, and establishes a conflict risk scheduling coefficient;

[0057] The concurrent task optimization module is based on the conflict risk scheduling coefficient, combines the file transfer request and the node load status data, analyzes the distribution law of the node task execution path, calculates the optimization value of the relationship between the file transfer path and the node task, adjusts the file transfer path and reallocates the node tasks, and establishes an optimized task distribution rate.

[0058] Compared with the prior art, the advantages and positive effects of the present invention are:

[0059] In the present invention, through path feature extraction and file exchange sequence analysis, critical paths are accurately identified and task priorities are dynamically marked, thereby improving the scientific nature of task scheduling and resource utilization efficiency. Combined with task status monitoring, dependent path weights and task priority markings are adjusted in real time to achieve flexibility and adaptability in task allocation and reduce task delays caused by resource competition. By recording request frequency and time period, resource allocation and short-term task sequence are optimized, high-frequency request resource conflicts are effectively alleviated, and resource allocation efficiency is improved. Through the analysis of node historical resource distribution data and time distribution trends, task distribution is adjusted in combination with differences, resource load is balanced, and the stability and balance of the system are enhanced. File transfer paths are optimized based on load status and task execution paths, and concurrent task distribution strategies are improved to improve file transfer efficiency and multi-task processing capabilities. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] Figure 1 It is a schematic diagram of the workflow of the present invention;

[0061] Figure 2 A flowchart of the steps for obtaining the task priority sequence value of the present invention;

[0062] Figure 3 A flow chart of the steps for obtaining the dynamic priority adjustment value of the present invention;

[0063] Figure 4 A flowchart of the steps for obtaining the short-term resource allocation result of the present invention;

[0064] Figure 5 A flowchart of the steps for obtaining the conflict risk scheduling sequence value of the present invention;

[0065] Figure 6 The present invention is a flowchart of the steps for obtaining the task distribution rate optimization. DETAILED DESCRIPTION

[0066] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0067] In the description of the present invention, it should be understood that the terms "length", "width", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside" and the like indicate positions or positional relationships based on the positions or positional relationships shown in the drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention. In addition, in the description of the present invention, "multiple" means two or more, unless otherwise clearly and specifically defined.

[0068] Embodiment 1

[0069] See also Figure 1 The present invention provides a technical solution: a distributed file processing method, comprising the following steps:

[0070] S1: Based on the task allocation node and data exchange interface, extract the path characteristics of the task input and output files, analyze the exchange order of the file storage path, calculate the topological structure and length of the path, filter the key path according to the path length and mark the task priority, and generate the task priority sequence value in order;

[0071] S2: Based on the task priority sequence value and the task status monitoring data of the resource manager, the file dependency path changes after the task is completed are analyzed, the weights of the dependency paths are recalculated, the task priority tags are updated according to the weight changes, the association between the paths and nodes is adjusted, and a dynamic priority adjustment value is generated;

[0072] S3: Based on the dynamic priority adjustment value, the storage nodes are screened according to the request frequency, the time period records of conflicting resources are extracted, the resource request frequencies are compared and the conflicting status is marked, priority resources are allocated to the conflicting tasks, the execution order of short-term tasks is adjusted, and the short-term resource allocation results are generated;

[0073] S4: Based on the short-term resource allocation results and node load parameters, extract historical storage distribution data, analyze the time distribution trend of resource usage, calculate the difference between resource status and current distribution, adjust the task distribution order according to the conflict trend, and generate the conflict risk scheduling order value;

[0074] S5: Based on the conflict risk scheduling order value, combined with the file transfer request and the load status of the resource manager, extract high-load storage nodes and tasks, analyze the execution path of node tasks, adjust the relationship between file transfer path and node tasks, optimize the resource distribution order of concurrent tasks, and calculate the optimized task distribution rate.

[0075] The task priority sequence value includes the topological structure of the task path, path length, critical path mark, and task priority mark. The dynamic priority adjustment value specifically includes the dependent path weight, task priority mark adjustment value, and the path-node association relationship. The short-term resource allocation result includes the conflict resource time period record, resource request frequency, conflict status mark, and short-term task execution order. The conflict risk scheduling sequence value includes node load parameters, resource time distribution trend, resource distribution difference, and conflict trend adjustment results.

[0076] See also Figure 2 , the specific steps for obtaining the task priority order value are:

[0077] S101: extracting input and output file path features of multiple nodes in the task path, calculating the path topology structure according to the uniqueness and hierarchical relationship of the path, marking the topology structure in layers according to the relationship between the path nodes, counting the connection relationship of each layer of path nodes and the number of associated file exchanges, and generating hierarchical features of the path;

[0078] The uniqueness information of the path is extracted as the path mark, and the hierarchical relationship of the path is analyzed layer by layer. The correlation between the layers is determined according to the connection order of the nodes. The hierarchical model of the path is established through the distribution relationship between the number of path nodes and their topological structure. The number of file exchanges and the number of nodes in each level are counted. The number of file exchanges and the number of nodes are used as quantitative parameters of the hierarchical relationship. The characteristic relationship of each level is marked according to the quantitative parameter results to generate the hierarchical characteristics of the path.

[0079] S102: performing file exchange sequence analysis on the hierarchical features of the path, sorting and counting the exchange sequence according to the time sequence of file exchange between nodes, counting the path coverage and the uniformity of the coverage based on the arrangement results of the file exchange sequences of multiple paths, calculating the length feature of the path, and generating the path length feature;

[0080] According to the topological sorting of the nodes in each path, the file exchange timing information is extracted in sequence, the frequency and order of file transfer in the path are counted, the timing distribution of each file exchange is extracted and the file exchange sequence is generated through statistical tools, the path coverage is calculated according to the order difference of the file exchange sequence, the uniformity information of the path coverage is extracted, the impact of the coverage on the path integrity is analyzed in combination with the length difference in the path, and the path length feature is generated using the path coverage and length difference.

[0081] S103: Screen the path length features, remove the paths with lengths lower than the global average length by combining the length feature values ​​of multiple paths, and calculate the comprehensive weighted lengths of the remaining paths using the formula:

[0082] ;

[0083] The path with the largest weight length value among the screening paths is marked as the critical path;

[0084] in, Represents the comprehensive weight length of the path, Represents the path The length of the segment, represents the weight of the road segment length, is the path criticality adjustment coefficient, Represents the maximum length of multiple segments in the path, Represents the total number of segments in the path, is the node level adjustment coefficient;

[0085] formula:

[0086] ;

[0087] The benefit of the formula is that it calculates the overall weight of the path by introducing the product of the path segment length and the weight, and combines the path maximum length adjustment coefficient and the node level coefficient to comprehensively reflect the importance and complexity of the path.

[0088] Detailed explanation of the formula and the process of formula calculation and derivation:

[0089] in, Represents the path The length of the segment is calculated by the straight-line distance between the path nodes. Represents the path The weight of the segment is quantified based on the combined weight of the importance of the path segment and the frequency of file exchange. is the path importance adjustment coefficient, which is obtained by quantifying the importance weight of the path segment to the critical path. is the node level adjustment coefficient, which is obtained by statistically quantifying the topological level and level distribution of the nodes. Indicates the total number of segments in the path, which is obtained by directly counting the number of all segments in the path. Indicates the maximum length of each segment in the path, which is obtained by counting the maximum value of all path segment lengths.

[0090] The calculation is performed using the following parameter values:

[0091]

[0092] The calculation process is as follows:

[0093]

[0094] The result shows that the comprehensive weight length value is 8.17, which is significantly higher than the global average length, indicating that the path is of high importance and should be marked as a critical path.

[0095] S104: Assigning task priorities to the critical path, prioritizing nodes according to the hierarchical relationship of nodes in the path topology structure, adjusting the node order in combination with the timing of the file exchange sequence, and generating a task priority sequence value based on the adjusted node order.

[0096] Extract the hierarchical relationship of nodes in the path topology structure, count the number of nodes in each level and calculate the hierarchical depth of the nodes through the direct association relationship between the nodes, adjust the priority between the nodes layer by layer according to the timing information of the file exchange order, and perform weighted sorting of the node priorities according to the hierarchical depth. Quantify the results of the weighted sorting into the priority sequence value of the task, and generate the task priority sequence value.

[0097] See also Figure 3 , the specific steps for obtaining the dynamic priority adjustment value are:

[0098] S201: extracting dependent path change information after task completion based on task priority order values ​​and task status monitoring data provided by the resource manager, performing preliminary analysis on the dependent paths through task completion time between nodes, calculating path weight changes, and generating preliminary dependent path weights;

[0099] First, according to the updated data of task completion time, its impact on the path topology structure is analyzed node by node, the task completion time difference and path span are calculated, and the change results are mapped to the path nodes one by one according to the path association relationship. Secondly, combined with the task completion time difference between nodes, the preliminary weight of each path is calculated, and the influence of the mean and maximum value of the task completion time on the weight is considered in a weighted manner. Then, the preliminary dependent path weight is generated by the accumulated value of the weight nodes in the path. Finally, the rationality of the preliminary weight is checked through the integrity of the path topology adjustment to ensure that the weight calculation result is consistent with the node association change, and the preliminary dependent path weight is generated.

[0100] S202: Recalculate the initial dependency path weights, taking into account task urgency, resource utilization efficiency, and delay time, using the formula:

[0101] ;

[0102] Generate updated dependency path weights;

[0103] in, represents the updated path weight, represents the original path weight, Represents resource utilization efficiency, Represents the task delay time, is the time after the task is completed. is the time decay constant, Indicates the number of tasks in the dependency path;

[0104] formula:

[0105] ;

[0106] The usefulness of the formula is that by introducing the time decay factor , combined with resource efficiency With delay time ,The dynamic nature of path weight calculation is improved and the model is more sensitive to time and resource changes.

[0107] Detailed explanation of the formula and the process of formula calculation and derivation:

[0108] (Original path weight, collected through the initial association weights of path nodes), (Resource utilization efficiency, obtained by monitoring the ratio of node resource usage to available resources), (Task delay time, obtained by subtracting the difference between the actual completion time of the task and the scheduled completion time), (The time after the task is completed, based on the path adjustment trigger time record), (time decay constant, combined with path adjustment stability and node update frequency setting), put it into the formula:

[0109]

[0110]

[0111]

[0112] The calculated path update weight is 2.4561. This result shows that the weight of the updated path is lower than the initial weight, indicating that the resource utilization efficiency is high and the delay time has little impact. The result can be used for further task priority adjustment.

[0113] S203: According to the updated dependent path weights, the changes in the task priority tags are counted, and a tag adjustment basis is generated by comparing the resource allocation amount and the task completion rate of the node, and the task priorities are reallocated to generate the task priorities before and after the adjustment;

[0114] First, the distribution of node resource allocation within the path is compared to analyze the impact of the updated weight on the task mark. Secondly, based on the updated weight results, the priority marks of the task nodes are reallocated, the difference between the task completion time and the priority mark is compared, and the weight influence factor of the priority mark is adjusted based on the maximum and mean resource consumption. Finally, a statistical comparison value of the priority mark before and after the adjustment is generated to confirm whether the updated mark is reasonable and generate a comparison of the task priorities before and after the adjustment.

[0115] S204: Based on the changes in task priorities and path weights before and after the adjustment, the path association relationship of the task nodes is adjusted, the path order relationship of the nodes is reconfigured according to the time sensitivity of the task, and a dynamic priority adjustment value is generated.

[0116] First, the time sensitivity changes of the task path are analyzed, and the path priority is dynamically adjusted according to the time span and resource allocation between nodes. Secondly, the association order of the path is reordered through the node with the highest time sensitivity, and the weight distribution differences within the associated path are analyzed to determine the connectivity changes of the path adjustment. Finally, the adjusted path order relationship is generated, and the dynamic priority adjustment value is generated based on the above results to ensure that the adjustment value reflects the association dynamics after the path update.

[0117] See also Figure 4 , the specific steps for obtaining the short-term resource allocation results are:

[0118] S301: extracting the request frequency based on the dynamic priority adjustment value, counting the number of accesses to the resource node in the request cycle, calculating the average request frequency according to the total number of requests for each resource node, screening resource nodes with high request frequencies by comparing the average request frequencies of the nodes, and generating a list of high-frequency request resource nodes;

[0119] By analyzing the request data of each resource node, the node request data is grouped according to time periods, the number of accesses in each time period is counted, and the average request frequency of each node is calculated. The frequency fluctuation of the node in each time period is calculated based on the average request frequency, and the nodes with higher average request frequencies are screened out. The higher frequency is judged by comprehensively judging the statistical value and distribution of the total number of node requests. The specific steps are as follows: the first step is to extract the request time and request number of the node, and divide the time into several time periods by hour. For example, 24 hours is divided into 24 segments of 1 hour each, and the number of access requests in each time period is summarized to form the initial request frequency distribution. The second step is to calculate the total number of requests and distribution range of each node, and use the formula Calculate the average request frequency, where For the Node frequency,In the third step, a filtering threshold is set based on the average request frequency,,and nodes that exceed the global average frequency are marked as,high-frequency nodes.,After filtering, the high-frequency nodes record their request distribution data,,and generate a list of high-frequency request resource nodes.

[0120] S302: Extract the time period records of conflicting resources according to the high-frequency request resource node list, count the concurrent requests for each resource in the conflicting time period, and calculate the priority allocation weight of the conflicting resources using the formula:

[0121] ;

[0122] Generate a list of conflicting resource allocation weights;

[0123] in, represents the resource allocation weight, Indicates The request intensity of the resource, Indicates the request frequency, Indicates the number of concurrent conflicts, is the conflict impact adjustment coefficient, represents the global mean of the number of conflicts, Indicates the total number of conflicting resources;

[0124] formula:

[0125] ;

[0126] The benefit of the formula is that it improves the nonlinear effect of request frequency and intensity on weight by introducing square calculation, and introduces the global mean of conflict number Adjusted the dynamic regulation of the number of conflicts on the weights, enhancing the adaptability of the formula in complex conflict scenarios;

[0127] Detailed explanation of the formula and the process of formula calculation and derivation:

[0128] Assume there are 3 conflicting resources, record them separately The number of conflicts is , conflict adjustment coefficient , global conflict mean , first calculate the numerator part ,have to

[0129] Then calculate the denominator ,have to ;

[0130] Dividing the numerator and denominator, we get ;

[0131] Finally, sum and square each term to get ;

[0132] Generate a conflict resource allocation weight list, which shows that the weight value directly reflects the priority of resource allocation and is used for subsequent resource status adjustment and task allocation.

[0133] S303: adjusting the conflicting resource status mark according to the conflicting resource allocation weight list, allocating priority resources according to the weight value, reordering the priority allocated resource nodes in combination with the conflicting time period, adjusting the execution order of the short-term tasks, and generating a priority allocation task order;

[0134] The allocation weight of each resource node is compared with the priority of the task, and the task order is readjusted based on the priority allocation weight value. The specific execution process is as follows: first, the resource node with the highest allocation weight value is extracted, and the node is matched with the conflicting items in the task priority order. The task order is rearranged by matching the task with the highest resource priority, and the task resources are allocated according to the distribution time period of the node resources. The adjusted task order is optimized in combination with the node load rate within the time period, and the execution order of short-term tasks is adjusted to generate a priority allocation task order.

[0135] S304: Recalculate the resource utilization rate of the short-term task according to the priority order of task allocation, optimize the task allocation plan in combination with the current load of the resource node, adjust the task allocation logic according to the relationship between the resource utilization rate and the current load, and generate a short-term resource allocation result.

[0136] Extract the adjusted task resource allocation, and calculate the amount of resources allocated to each task in combination with the resource node load. The specific steps are as follows: the first step is to count the resource usage of each node assigned task, based on the amount of resources required for task execution, combined with the current load rate of the node to calculate the remaining available resources; the second step is to group the node load rate and the remaining available resources according to the execution time period, calculate the dynamic changes of the node load in different time periods, and optimize the task allocation order again through the load changes between node groups. The optimized resource utilization rate is combined with the task completion rate to calculate its total efficiency and generate the short-term resource allocation result.

[0137] See also Figure 5 ,The specific steps for obtaining the conflict risk scheduling order value are:

[0138] S401: extracting node resource distribution data based on the short-term resource allocation result and the node load parameter, determining the load distribution difference of multiple nodes by comparing and analyzing the resource usage status of the node and the current load value, and generating a node load difference analysis result;

[0139] By monitoring the resource usage of each node in different time periods, combining the actual load of the node, and counting the frequency of resource allocation, the resource usage rate of each node per unit time is calculated as the resource distribution ratio. ,in Indicates The resource distribution ratio of the node, is the actual usage of node resources, is the total resource capacity of the node. After counting the resource occupancy ratio of the node, a comparative analysis is performed. By comparing the difference between the current load ratio of each node and the average load level, the load difference of the node is further calculated. The formula ,in It represents the average value of node resource utilization. The analysis results show that nodes with larger node load differences are more likely to conflict in resource distribution, generating node load difference analysis results.

[0140] S402: Calculate the time distribution trend of resource usage based on the node load difference analysis result, evaluate the resource usage fluctuation in the time period by counting the resource allocation frequency and node load changes in multiple time periods, identify the key time period, and generate a resource usage time distribution trend report;

[0141] By statistically analyzing the fluctuations in the node resource allocation ratio in different time periods, the average resource usage in each time period is extracted, and the fluctuation range is evaluated by calculating the change rate of resource usage in each time period, and the threshold is set. , when the resource distribution difference rate When the resource usage data of each key period is combined, the total usage is calculated by summing up the resource usage data of all nodes in the key period, thereby generating a resource usage time distribution trend. Finally, the high fluctuation interval in the trend is marked as a resource usage key time period, and a resource usage time distribution trend report is generated.

[0142] S403: Analyze the resource usage time distribution trend report, calculate the difference between the resource status and the current distribution, and quantitatively evaluate the resource distribution difference and conflict risk using the formula:

[0143] ;

[0144] Calculate the resource distribution difference and generate resource status and distribution difference evaluation results;

[0145] in, Indicates the resource distribution difference, Indicates Resource usage status for a time period, Indicates the average value of resource usage status. Indicates the standard deviation of resource usage status, Indicates the total number of time periods;

[0146] formula:

[0147] ;

[0148] The benefit of the formula is that by indexing resource distribution deviations, it can enhance sensitivity to extreme differences, thereby accurately identifying time periods with high conflict risks and prioritizing them.

[0149] Detailed explanation of the formula and the process of formula calculation and derivation:

[0150] Indicates The resource usage status of each time period is obtained by monitoring the resource usage ratio of each time period. , calculate the mean value for each time period , calculate the standard deviation , put the data into the formula:

[0151] ;

[0152] The result shows that the resource distribution difference in the current time period is low and has not reached a high-risk state. However, in actual scenarios, it is necessary to combine larger-scale time period data to obtain accurate distribution difference assessment results.

[0153] S404: Based on the resource status and distribution difference evaluation results, the distribution order of tasks is adjusted by calculating the conflict trend, the matching degree between tasks and resources is optimized, the adjusted task distribution order is generated, and the conflict risk scheduling order value is obtained.

[0154] By analyzing the resource distribution deviation value, the task distribution order is adjusted according to the resource load of different nodes, and the priority allocation weight of each task is calculated. ,in represents the task allocation weight, Indicates the resource requirements of the task, It represents the deviation value of the resource distribution of the task. By calculating the weight of each task and sorting them to determine the priority, the adjusted task distribution order is generated, and the conflict risk scheduling order value is generated based on the order.

[0155] See also Figure 6 ,The specific steps for obtaining the optimized task distribution rate are:

[0156] S501: Based on the conflict risk scheduling order value, combined with the file transfer request and the load status of the resource manager, the nodes and associated tasks with uneven resource distribution are analyzed, and by comparing the resource distribution and load of the nodes and tasks, the resource distribution deviation data is extracted to generate a resource uneven distribution report;

[0157] First, data on the resource distribution and load status of each node is extracted. The extracted data includes the node's task occupancy rate, resource distribution density, and file transfer delay. By comparing the node's task occupancy rate with the load threshold, the nodes with task occupancy rates higher than the average are screened out. Then, the resource distribution unevenness rate of each node is calculated. The resource distribution unevenness rate is obtained by statistically analyzing the task execution time and the standard deviation of the task's occupied resources. The calculation formula is: ,in The amount of resources allocated to a node in a certain period of time. is the mean value of resource allocation for the node in all time periods. The total number of time periods is further analyzed to match the source and target nodes of the file transfer. The average delay of the file transfer path is calculated to find the file transfer path with an average delay greater than the preset threshold. The resource uneven distribution report is generated based on the calculation results.

[0158] S502: Analyze the execution path of the node task according to the resource uneven distribution report, identify the execution efficiency bottleneck caused by the uneven resource distribution by comparing and calculating the relationship between the execution timing of the node task and the resource distribution, extract the task optimization requirements, and generate the node task execution path analysis results;

[0159] First, the execution timing of each path is counted, the start and end time of the node task is recorded by timestamp, and the maximum number of concurrent tasks in the path is calculated by formula Get the concurrent utilization of the path, where It is The task duration of each node, is the number of path nodes, The total duration of the path is then analyzed. The total resource allocation ratio of the path is obtained by accumulating the node resource allocation ratio. Combined with the execution efficiency index of the path, high-latency and low-efficiency paths are extracted and the node task execution path analysis results are generated.

[0160] S503: Based on the analysis result of the node task execution path, the relationship between the file transfer path and the node task is adjusted by reconfiguring the file transfer path, using the formula:

[0161] ;

[0162] Reallocate node resources and optimize task paths, and calculate and generate the optimized task distribution rate;

[0163] in, represents the optimization task distribution rate, It is The resource usage of each task, represents the average resource usage of all tasks, represents the standard deviation of resource utilization, Represents the total number of tasks, indicating the scope of statistics and calculations on resource usage of all tasks.

[0164] formula:

[0165] ;

[0166] The formula is beneficial in that it effectively enhances the evaluation of task distribution balance and optimizes the task path allocation logic by normalizing the resource usage deviation of each task and introducing the trade-off factor of the mean and standard deviation.

[0167] Detailed explanation of the formula and the process of formula calculation and derivation:

[0168] Get node task resource usage , calculate the average , standard deviation , put it into the formula:

[0169] ;

[0170] ;

[0171] The result shows that the optimization task distribution rate is 63.33%, and the task distribution balance is low, indicating that resource allocation needs to be further adjusted to improve distribution efficiency and path optimization effect.

[0172] A distributed file processing system, the distributed file processing system is used to execute the above-mentioned distributed file processing method, the system comprises:

[0173] The task priority analysis module extracts the file exchange order in the path based on the path characteristics of the task input and output files, calculates the path topology structure and length, marks the priority of the tasks in the path, arranges the priorities and generates the path priority sequence value;

[0174] The dynamic path adjustment module calculates the weight change of the dependent path after the task is completed based on the path priority sequence value and the task status data of the resource node, compares the path weight values ​​before and after the update, re-marks the task priority and adjusts the node association information, and establishes dynamic priority adjustment parameters;

[0175] The short-term resource scheduling module adjusts parameters based on dynamic priority, extracts the request frequency data and conflict time period records of resource nodes, calculates the resource request frequency within the time period and marks the conflict status, allocates priority resources and adjusts the execution order of short-term tasks, calculates the resource distribution difference value of task allocation, and generates the short-term resource distribution adjustment rate;

[0176] The conflict risk assessment module extracts the distribution data and load parameters of resource nodes based on the short-term resource distribution adjustment rate, calculates the resource time distribution trend value and compares it with the current distribution trend value, identifies resource conflict trend changes, analyzes task distribution sequence differences and adjusts node associations, and establishes a conflict risk scheduling coefficient;

[0177] The concurrent task optimization module is based on the conflict risk scheduling coefficient, combines the file transfer request and the node load status data, analyzes the distribution law of the node task execution path, calculates the optimization value of the relationship between the file transfer path and the node task, adjusts the file transfer path and reallocates the node tasks, and establishes an optimized task distribution rate.

[0178] The above are only preferred embodiments of the present invention and are not intended to limit the present invention in other forms. Any technician familiar with the profession may use the technical contents disclosed above to change or modify them into equivalent embodiments with equivalent changes and apply them to other fields. However, any simple modification, equivalent change and modification made to the above embodiments based on the technical essence of the present invention without departing from the technical solution of the present invention still falls within the protection scope of the technical solution of the present invention.

Claims

1. A distributed file processing method, characterized in that: The following steps are involved: S1: Based on the task allocation node and data exchange interface, extract the path characteristics of the task input and output files, analyze the exchange order of the file storage path, calculate the topological structure and length of the path, filter the key path according to the path length and mark the task priority, and generate the task priority sequence value in order; S2: Based on the task priority sequence value and in combination with the task status monitoring data of the resource manager, analyze the file dependency path changes after the task is completed, recalculate the dependency path weights, update the task priority mark according to the weight changes, adjust the association between the path and the node, and generate a dynamic priority adjustment value; S3: Based on the dynamic priority adjustment value, filter the storage nodes according to the request frequency, extract the time period records of the conflicting resources, compare the resource request frequencies and mark the conflicting states, allocate priority resources to the conflicting tasks, adjust the execution order of the short-term tasks, and generate the short-term resource allocation results; S4: Based on the short-term resource allocation result and node load parameters, extract historical storage distribution data, analyze the time distribution trend of resource usage, calculate the difference between resource status and current distribution, adjust the task distribution order according to the conflict trend, and generate a conflict risk scheduling order value; S5: Based on the conflict risk scheduling order value, combined with the file transfer request and the load status of the resource manager, extract the high-load storage nodes and tasks, analyze the execution path of the node tasks, adjust the relationship between the file transfer path and the node tasks, optimize the resource distribution order of the concurrent tasks, and calculate the optimized task distribution rate; The task priority sequence value includes the topological structure of the task path, the path length, the critical path mark, and the task priority mark. The dynamic priority adjustment value is specifically the dependent path weight, the task priority mark adjustment value, and the path-node association relationship. The short-term resource allocation result includes the conflict resource time period record, the resource request frequency, the conflict status mark, and the short-term task execution order. The conflict risk scheduling sequence value includes the node load parameter, the resource time distribution trend, the resource distribution difference, and the conflict trend adjustment result. The steps for obtaining the optimization task distribution rate are specifically as follows: S501: Based on the conflict risk scheduling order value, combined with the file transfer request and the load status of the resource manager, analyzing the nodes and associated tasks with uneven resource distribution, extracting resource distribution deviation data by comparing the resource distribution and load of the nodes and tasks, and generating a resource uneven distribution report; S502: Analyze the execution path of the node task according to the uneven resource distribution report, identify the execution efficiency bottleneck caused by uneven resource distribution by comparing and calculating the relationship between the execution timing of the node task and the resource distribution, extract task optimization requirements, and generate the node task execution path analysis result; S503: Based on the analysis result of the node task execution path, the relationship between the file transmission path and the node task is adjusted by reconfiguring the file transmission path, using the formula: ； Reallocate node resources and optimize task paths, and calculate and generate the optimized task distribution rate; in, represents the optimization task distribution rate, It is The resource usage of each task, represents the average resource usage of all tasks, represents the standard deviation of resource utilization, Represents the total number of tasks, indicating the scope of statistics and calculations on resource usage of all tasks.

2. The distributed file processing method according to claim 1, characterized in that: The steps for obtaining the task priority order value are specifically as follows: S101: extracting input and output file path features of multiple nodes in the task path, calculating the path topology structure according to the uniqueness and hierarchical relationship of the path, marking the topology structure in layers according to the relationship between the path nodes, counting the connection relationship of each layer of path nodes and the number of associated file exchanges, and generating hierarchical features of the path; S102: performing file exchange sequence analysis on the hierarchical features of the path, sorting and counting the exchange sequence according to the time sequence of file exchange between nodes, counting the path coverage and the uniformity of the coverage based on the arrangement results of the file exchange sequences of multiple paths, calculating the length feature of the path, and generating the path length feature; S103: Screen the path length features, remove the paths with lengths shorter than the global average length by combining the length feature values ​​of multiple paths, and calculate the comprehensive weighted lengths of the remaining paths using the formula: ； The path with the largest weight length value among the screening paths is marked as the critical path; in, Represents the comprehensive weight length of the path, Represents the path The length of the segment, represents the weight of the road segment length, is the path criticality adjustment coefficient, Represents the maximum length of multiple segments in the path, Represents the total number of segments in the path, is the node level adjustment coefficient; S104: Assigning task priorities to the critical path, prioritizing nodes according to the hierarchical relationship of nodes in the path topology structure, adjusting the node order in combination with the timing of the file exchange sequence, and generating a task priority sequence value based on the adjusted node order.

3. The distributed file processing method according to claim 2, characterized in that: The steps for obtaining the dynamic priority adjustment value are specifically as follows: S201: extracting dependent path change information after task completion based on the task priority sequence value and task status monitoring data provided by the resource manager, performing a preliminary analysis on the dependent path through the task completion time between nodes, calculating the path weight change, and generating a preliminary dependent path weight; S202: recalculate the preliminary dependency path weights, comprehensively consider task urgency, resource utilization efficiency and delay time, and use the formula: ； Generate updated dependency path weights; in, represents the updated path weight, represents the original path weight, Represents resource utilization efficiency, Represents the task delay time, is the time after the task is completed. is the time decay constant, Indicates the number of tasks in the dependency path; S203: According to the updated dependent path weights, count the changes in the task priority tags, generate a tag adjustment basis by comparing the resource allocation amount and task completion rate of the node, and reallocate the task priorities to generate the task priorities before and after the adjustment; S204: adjusting the path association relationship of the task nodes in combination with the task priorities and path weight changes before and after the adjustment, reconfiguring the path order relationship of the nodes according to the time sensitivity of the tasks, and generating a dynamic priority adjustment value.

4. The distributed file processing method according to claim 3, characterized in that: The steps for obtaining the short-term resource allocation result are specifically as follows: S301: extracting the request frequency based on the dynamic priority adjustment value, counting the number of accesses to the resource node within the request cycle, calculating the average request frequency according to the total number of requests for each resource node, screening resource nodes with high request frequencies by comparing the average request frequencies of the nodes, and generating a list of high-frequency request resource nodes; S302: extract the time period records of conflicting resources according to the high-frequency request resource node list, count the concurrent requests for each resource in the conflicting time period, and calculate the priority allocation weight of the conflicting resources using the formula: ； Generate a list of conflicting resource allocation weights; in, represents the resource allocation weight, Indicates The request intensity of the resource, Indicates the request frequency, Indicates the number of concurrent conflicts, is the conflict impact adjustment coefficient, represents the global mean of the number of conflicts, Indicates the total number of conflicting resources; S303: adjusting the conflicting resource status mark according to the conflicting resource allocation weight list, allocating priority resources according to the weight values, reordering the priority allocated resource nodes in combination with the conflicting time periods, adjusting the execution order of the short-term tasks, and generating a priority allocation task order; S304: recalculate the resource utilization rate of the short-term task according to the priority task allocation order, optimize the task allocation plan in combination with the current load of the resource node, adjust the task allocation logic according to the relationship between the resource utilization rate and the current load, and generate a short-term resource allocation result.

5. The distributed file processing method according to claim 4, characterized in that: The steps for obtaining the conflict risk scheduling order value are specifically as follows: S401: extracting node resource distribution data based on the short-term resource allocation result and the node load parameter, determining the load distribution difference of multiple nodes by comparing and analyzing the resource usage status of the node and the current load value, and generating a node load difference analysis result; S402: Calculate the time distribution trend of resource usage according to the node load difference analysis result, evaluate the resource usage fluctuation of the time period by counting the resource allocation frequency and node load changes in multiple time periods, identify the key time period, and generate a resource usage time distribution trend report; S403: Analyze the resource usage time distribution trend report, calculate the difference between the resource status and the current distribution, and quantitatively evaluate the resource distribution difference and conflict risk using the formula: ； Calculate the resource distribution difference and generate resource status and distribution difference evaluation results; in, Indicates the resource distribution difference, Indicates Resource usage status for a time period, Indicates the average value of resource usage status. Indicates the standard deviation of resource usage status, Indicates the total number of time periods; S404: Based on the resource status and distribution difference evaluation results, the distribution order of tasks is adjusted by calculating the conflict trend, the matching degree between tasks and resources is optimized, an adjusted task distribution order is generated, and a conflict risk scheduling order value is obtained.

6. A distributed file processing system, characterized in that: According to any one of claims 1 to 5, the distributed file processing method comprises: The task priority analysis module extracts the file exchange order in the path based on the path characteristics of the task input and output files, calculates the path topology structure and length, marks the priority of the tasks in the path, arranges the priorities and generates the path priority sequence value; The dynamic path adjustment module calculates the weight change of the dependent path after the task is completed based on the path priority sequence value and the task status data of the resource node, compares the path weight values ​​before and after the update, re-marks the task priority and adjusts the node association information, and establishes dynamic priority adjustment parameters; The short-term resource scheduling module extracts the request frequency data and conflict time period records of the resource nodes based on the dynamic priority adjustment parameters, calculates the resource request frequency within the time period and marks the conflict status, allocates priority resources and adjusts the execution order of short-term tasks, calculates the resource distribution difference value of the task allocation, and generates the short-term resource distribution adjustment rate; The conflict risk assessment module extracts the distribution data and load parameters of resource nodes based on the short-term resource distribution adjustment rate, calculates the resource time distribution trend value and compares it with the current distribution trend value, identifies resource conflict trend changes, analyzes task distribution sequence differences and adjusts node associations, and establishes a conflict risk scheduling coefficient; The concurrent task optimization module is based on the conflict risk scheduling coefficient, combines the file transfer request and the node load status data, analyzes the distribution law of the node task execution path, calculates the optimization value of the relationship between the file transfer path and the node task, adjusts the file transfer path and reallocates the node tasks, and establishes an optimized task distribution rate.

Images