Large model parallel computing acceleration method and system applied to multi-node cluster

Abstract

The invention provides a large-model parallel computing acceleration method and system applied to a multi-node cluster, and belongs to the technical field of artificial intelligence computing, and the method comprises the steps: firstly generating a demand feature set containing a large-model overall computing task resource consumption trend, a data dependence mode and a logic connection requirement; recombining node resource configuration of the multi-node cluster based on the demand feature set; splitting the calculation task into a plurality of task segments according to the demand feature set, and allocating the task segments to corresponding node resource configurations; constructing a task fragment data circulation channel based on data dependency mode setting parameters; finally, task fragment processing results of all the nodes are received, correlation is analyzed according to logic connection requirements, and a final acceleration result is generated. The performance and efficiency of processing large-model calculation tasks through a multi-node cluster are effectively improved.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence computing technology, and more specifically, to a method and system for accelerating parallel computing of large models applied to multi-node clusters. Background Technology

[0002] With the rapid development of artificial intelligence technology, large models have been widely used in many fields such as natural language processing and computer vision. Large models typically contain massive amounts of parameters, and their computational tasks are extremely complex and involve huge amounts of data, making computation on a single node extremely difficult, or even impossible. Therefore, multi-node cluster parallel computing has become the main way to handle large model computational tasks.

[0003] However, existing multi-node cluster parallel computing methods have several problems. Firstly, they lack a precise understanding of the characteristics of large-scale model computation tasks in resource allocation, often employing a rather coarse-grained strategy. This fails to dynamically adjust node resources based on the resource consumption trends of computational tasks, resulting in some nodes being idle while others are overloaded, leading to low overall resource utilization. Secondly, task splitting and allocation do not adequately consider the data dependency patterns and logical connection requirements of computational tasks, causing disordered execution and data processing sequences of task fragments across different nodes. This increases data transmission and synchronization overhead, reducing the efficiency of parallel computing. Furthermore, the construction of data flow channels lacks specificity, failing to optimize channel parameters based on data dependency patterns, further impacting computational performance. Summary of the Invention

[0004] In view of the aforementioned problems, and in conjunction with the first aspect of the present invention, embodiments of the present invention provide a method for accelerating large-scale parallel computing of multi-node clusters, the method comprising: Generate a set of requirements features for the overall computational task of the large model, which includes the resource consumption trend, data dependency pattern and logical connection requirements of the overall computational task of the large model. The node resource configuration of a multi-node cluster is reorganized based on the demand feature set. The node resource configuration is formed by adapting and combining the processor resources, memory resources and transmission resources of each computing node in the multi-node cluster according to the demand features in the demand feature set. Based on the resource consumption trends and logical connection requirements of the demand characteristics, the overall computational task of the large model is divided into multiple task segments, and each task segment is assigned to the corresponding node resource configuration. Construct a data flow channel between task fragments and node resource configurations, wherein the parameters of the data flow channel are set based on the data dependency mode of the task fragments; Receive the task fragment processing results output by the resource configuration of each node, analyze the correlation between the task fragment processing results based on the logical connection requirements of the demand feature set, and generate the final accelerated result of parallel computing of the large model.

[0005] In another aspect, embodiments of the present invention also provide a large-scale parallel computing acceleration system for multi-node clusters, including a processor and a machine-readable storage medium connected to the processor. The machine-readable storage medium is used to store programs, instructions, or code, and the processor is used to execute the programs, instructions, or code in the machine-readable storage medium to implement the above-described method.

[0006] Based on the above, this embodiment of the invention generates a large-scale model overall computation task requirement feature set that includes resource consumption trends, data dependency patterns, and logical connection requirements. Based on this feature set, it reorganizes the node resource configuration of a multi-node cluster. This dynamically adapts processor, memory, and transmission resources according to the actual needs of the computation task, effectively avoiding resource idleness and overload, and improving resource utilization. By splitting and allocating large-scale model computation tasks according to the requirement feature set, it fully considers the resource consumption and logical connection of tasks, making the execution of task segments between different nodes more reasonable and orderly, reducing data transmission and synchronization overhead. It constructs a data flow channel based on parameters set according to data dependency patterns, optimizing data transmission between nodes and further improving computational efficiency. Finally, based on logical connection requirements, it analyzes the correlation between the processing results of task segments, generating accurate final accelerated results for large-scale model parallel computation, significantly improving the performance and efficiency of multi-node clusters in processing large-scale model computation tasks. Attached Figure Description

[0007] Figure 1 This is a schematic diagram of the execution flow of the method for accelerating parallel computing of large models applied to multi-node clusters provided in the embodiments of the present invention.

[0008] Figure 2 This is a schematic diagram of exemplary hardware and software components of a large-scale parallel computing acceleration system for multi-node clusters provided in an embodiment of the present invention. Detailed Implementation

[0009] The present invention will now be described in detail with reference to the accompanying drawings. Figure 1 This is a flowchart illustrating a method for accelerating large-scale parallel computing in multi-node clusters, provided by an embodiment of the present invention. The following is a detailed description of this method for accelerating large-scale parallel computing in multi-node clusters.

[0010] Step S110: Generate a requirement feature set for the overall computational task of the large model, wherein the requirement feature set includes the resource consumption trend, data dependency pattern and logical connection requirements of the overall computational task of the large model.

[0011] In this embodiment, a large-scale model computation task is taken as an example. This task requires continuous processing of multiple types of input data and obtaining the final result through multiple related computational steps. When generating the requirement feature set, it is necessary to comprehensively analyze the resource usage, data flow relationships, and connection logic of each step during the execution of the task, extract information that reflects resource consumption trends, data dependency patterns, and logical connection requirements, and integrate the above information to form the requirement feature set.

[0012] Step S111: Analyze the execution flow of the overall computation task of the large model, and extract the processor resource usage time, memory resource usage and transmission resource usage frequency of each computation stage in the execution flow of the overall computation task of the large model, forming resource usage records for each resource type.

[0013] The overall computational task of this large model includes a data receiving stage, a data filtering stage, a feature transformation stage, a core computation stage, an intermediate result integration stage, a result optimization stage, and a final result output stage. When analyzing each computational stage, the time spent by the processor in data operations, logical processing, etc., in each stage needs to be recorded, for example, the processor's working time when filtering valid data in the data filtering stage.

[0014] Regarding memory resource usage, it is necessary to record the amount of memory space occupied by each step during operation for storing raw data, intermediate results, temporary variables, etc., such as the memory usage when storing transformed feature data in the feature transformation step.

[0015] The frequency of transmission resource usage needs to be recorded, specifically the number of times the transmission resources are invoked when data is transferred between different stages, such as the number of times the transmission resources are used when the core computing stage transfers computing data to the intermediate result integration stage.

[0016] The above information is organized into processor resources, memory resources, and transmission resources to form resource usage records for each resource type. Each record for each resource type corresponds to specific data for each computing stage.

[0017] Step S112: Perform time-series analysis on the resource usage records to generate a resource consumption change sequence for each computing stage within the execution cycle of the overall computing task of the large model. Based on the resource consumption change sequence, summarize the resource consumption trend of the overall computing task of the large model. The resource consumption trend reflects the changing pattern of resource demand of the overall computing task of the large model in different time periods.

[0018] A time-series analysis was performed on the resource usage records obtained above. The entire execution cycle of the overall computational task of the large model was used as the time frame, and multiple time windows were divided based on the execution characteristics of each computational stage. Within each time window, the processor resource usage time, memory resource usage, and transmission resource usage frequency of each computational stage were statistically analyzed.

[0019] Based on these statistics, the resource consumption changes of each computational stage under different time windows are generated, thus forming a resource consumption change sequence. This sequence shows that in the early stages of task execution, the resource consumption of data receiving and filtering stages is relatively high; as the task progresses, the resource consumption of feature transformation and core computation stages gradually increases and becomes the main component; in the later stages of the task, the resource consumption of intermediate result integration, result optimization, and final result output stages shows corresponding changes.

[0020] Based on these observations, the resource consumption trend of the overall computational task of the large model can be summarized, which can clearly reflect the changes in the task's demand for various resources at different times.

[0021] Step S1121: Divide the resource usage records into multiple time windows based on the overall execution time of the large model calculation task. The duration of each time window is set based on the execution rhythm of the calculation process, so that each time window contains complete resource usage data.

[0022] The time window is divided according to the execution rhythm of each computational stage in the large model. For example, the data receiving and data filtering stages have a faster execution rhythm, so the time window can be set to a shorter duration; the core computation stage and the intermediate result integration stage have a slower execution rhythm, so the time window is set to a longer duration.

[0023] The above settings ensure that each time window contains complete resource usage data for the corresponding computing stage. For example, during the execution period of the data receiving stage, time windows are divided into shorter durations, and each window can completely record the processor usage time, memory usage, and transmission resource usage frequency of that stage during that period. During the execution period of the core computing stage, time windows are divided into longer durations to fully cover all resource usage data for that stage during the corresponding period.

[0024] Step S1122: Calculate the processor resource usage time, memory resource usage, and transmission resource usage frequency for each computing stage within each time window to form resource statistics for each resource type within each time window.

[0025] For each defined time window, the resource usage of each computation stage is statistically analyzed. For processor resource usage, the total duration of continuous processor operation for each computation stage within that time window is calculated. For memory resource usage, the amount of memory space occupied by each computation stage within that time window is calculated. For transmission resource usage frequency, the number of times transmission resources are used when each computation stage transmits data with other stages within that time window is calculated.

[0026] The above statistical results are organized by time window and resource type to form resource statistics for each resource type within each time window. These data can clearly show the consumption of various resources in different computing stages within each time window.

[0027] Step S1123: Plot the resource consumption change curves for each calculation stage with the time window as the horizontal axis and the resource statistics data as the vertical axis. The resource consumption change curves for each calculation stage reflect the changes in resource consumption within different time windows.

[0028] Using the time window as the horizontal axis and the resource statistics within each time window as the vertical axis, resource consumption curves are plotted for each computation stage. For example, a curve showing the change in processor resource usage time is plotted for the data filtering stage, with the horizontal axis representing each time window and the vertical axis representing the processor resource usage time of the data filtering stage within the corresponding time window. Similarly, a curve showing the change in memory resource usage is plotted for the feature transformation stage, with the horizontal axis representing the time window and the vertical axis representing the memory resource usage of the feature transformation stage within the corresponding time window.

[0029] These curves can intuitively reflect the changes in the consumption of various resources by each computing stage within different time windows.

[0030] Step S1124: Extract change nodes from the resource consumption change curves of each calculation stage. Change nodes are the time windows in which resource statistics data show significant changes. Record the amount of resource data change corresponding to each change node.

[0031] We analyze the resource consumption curves of each computing stage to extract the change nodes. A change node refers to a point in time when the resource statistics show a significant change compared to the previous time window. For example, if the processor resource usage time of the core computing stage suddenly increases significantly in a certain time window, that time window is a change node.

[0032] For each change node, record the corresponding change in resource data, that is, the difference between the resource statistics data of this time window and the resource statistics data of the previous time window.

[0033] Step S1125: Based on the change nodes and resource data changes, generate a resource consumption change sequence for each computing stage. The resource consumption change sequence for each computing stage records the resource statistics data and change node information for each time window in chronological order.

[0034] By combining the extracted change nodes and their corresponding resource data changes, a resource consumption change sequence for each computational stage is generated. Within this sequence, resource statistics for each time window are recorded sequentially over time, with the location of each change node and its corresponding resource data change indicated.

[0035] For example, in the resource consumption change sequence of the data filtering process, the processor resource usage time of each window can be listed in the order of time windows, and the windows where obvious changes occur can be marked as change nodes, while the amount of resource data change at that node can be recorded.

[0036] Step S1126: Compare the resource consumption change sequences of different computing stages, identify computing stage groups with similar change patterns in the resource consumption change sequences of different computing stages, and analyze the common characteristics of resource consumption of computing stages within the computing stage group.

[0037] By comparing the resource consumption change sequences of different computing stages, computing stages with similar change patterns can be identified and grouped together to form computing stage groups. For example, the resource consumption change sequences of the data receiving stage and the data filtering stage are similar in terms of change trends and the distribution of change nodes, so they can be grouped into one computing stage group.

[0038] Analyze the resource consumption of each computational stage within this computational stage group, and summarize their common characteristics, such as the timing of peak resource consumption and the overall trend of resource demand.

[0039] Step S1127: Summarize the common characteristics of resource consumption of the computation links within the computation link group and the unique characteristics of resource consumption of individual computation links. Combine the time distribution within the execution cycle of the overall computation task of the large model to form the resource consumption trend of the overall computation task of the large model. The resource consumption trend of the overall computation task of the large model includes the main types of resources consumed and the scale of demand in different time stages of the overall computation task of the large model.

[0040] By summarizing the common characteristics within computational stages and the unique characteristics of individual computational stages, and combining this with the time distribution of the overall computational task execution cycle in the large model, resource consumption trends can be identified. For example, a certain computational stage group may have a higher demand for processor resources in the early stages of task execution, which is a common characteristic; while the core computational stage may have a uniquely high demand for memory resources in the middle stages of the task, which is a unique characteristic.

[0041] By combining these characteristics with the time distribution, we can identify the main types of resources consumed and the scale of demand for the task at different time stages. For example, in the early stage, the main resources consumed are processor resources, with a large demand scale; in the middle stage, the main resources consumed are memory resources, with a medium demand scale.

[0042] Step S1128: Associate the resource consumption trend of the overall computation task of the large model with the corresponding computation stage and the corresponding time window, and mark the impact range of the resource consumption trend of the overall computation task of the large model on the overall computation task of the large model.

[0043] The resulting resource consumption trends are correlated with corresponding computational stages and time windows to clarify which computational stages the trend involves and within which time windows it takes effect. For example, the initial resource consumption trend is mainly associated with the data receiving and data filtering stages, involving several time windows in the early stages of task execution.

[0044] At the same time, the impact of this resource consumption trend on the overall computing task of the large model is marked, such as the impact on the resource allocation strategy and execution efficiency of the task, as well as the specific impact on the execution progress of which links.

[0045] Step S113: Analyze the data transfer relationship between each computing stage, record the path, timing and amount of data transferred from one computing stage to another, form a data transfer graph based on the recorded data transfer relationship, extract the data dependency pattern of the overall computing task of the large model from the data transfer graph, and the data dependency pattern reflects the fixed pattern of data interaction between each computing stage.

[0046] Analyze the data transfer between each computation stage, record the specific path of data transfer from one stage to another, i.e., whether the data is transferred directly or through other intermediate nodes; the timing of the transfer, i.e., at what time or under what conditions the data is transferred, such as immediately after the previous stage is completed; and the amount of data transferred.

[0047] Based on the above records, a data transfer graph is formed, which graphically displays information such as the data flow path, transfer timing, and data volume between each stage. Data dependency patterns are extracted from the data transfer graph. These patterns reflect the fixed rules of data interaction between each computing stage. For example, a certain computing stage can only start execution after receiving specific data from the previous stage, and the amount of data transferred each time fluctuates within a certain range.

[0048] Step S114: Analyze the execution order of each calculation stage, determine the execution logic that must be followed between each calculation stage, record the correspondence between the calculation stages that need to be completed in advance and the calculation stages that are executed later, form a logical connection list, and extract the logical connection requirements of the overall calculation task of the large model from the logical connection list. The logical connection requirements reflect the constraints on the execution order between each calculation stage.

[0049] Analyze the execution order of each computational step to clarify the execution logic that must be followed between each step. For example, the data filtering step can only be executed after the data receiving step is completed, and the feature transformation step must begin after the data filtering step is completed.

[0050] Record the logical correspondence between the computational steps that need to be completed in advance and the computational steps that are executed later. For example, if the in advance is the data receiving step and the later step is the data filtering step, or if the in advance is the data filtering step and the later step is the feature transformation step, etc., to form a list of logical connections.

[0051] Extract the logical connection requirements from the logical connection list. These requirements reflect the constraints on the execution order between each computation stage, such as that a subsequent stage must wait for all its preceding stages to complete before it can start, and that parallel execution is not allowed.

[0052] Step S115: Integrate the resource consumption trend, data dependency pattern, and logical connection requirements, and mark the coverage and impact of each of the resource consumption trend, data dependency pattern, and logical connection requirements in the overall computational task of the large model to form a set of requirements features.

[0053] Integrate the resource consumption trends, data dependency patterns, and logical connection requirements obtained earlier. For resource consumption trends, mark the time stages they cover and the computational steps involved in the overall computational task of the large model, as well as the degree of impact on task execution, such as high, medium, and low.

[0054] For data dependency patterns, label the computational stages and data transfer scenarios they cover, as well as their impact on data flow efficiency. For logical connection requirements, label the execution order relationships they cover and their impact on the overall task flow.

[0055] The integrated information above is combined to form a set of requirements features, which comprehensively reflects the characteristics of the overall computing task of the large model in terms of resources, data, and logic.

[0056] Step S116: Supplement resource usage analysis for special computing links not covered in the resource consumption trend, and integrate the supplementary content into the requirement feature set; supplement data interaction records for uncertain transmission details in the data dependency pattern, and integrate the supplementary content into the requirement feature set; supplement execution logic descriptions for unclear constraints in the logical connection requirements, and integrate the supplementary content into the requirement feature set.

[0057] Examine resource consumption trends. If there are any uncovered special computing processes, such as some occasionally executed auxiliary computing processes, perform resource usage analysis on these processes, record data such as processor resource usage duration, memory resource usage, and transmission resource usage frequency, and integrate the above supplementary content into the requirement feature set.

[0058] Check the data dependency pattern. If there are any undetermined transmission details, such as unclear specific triggering conditions for certain data transmissions, supplement the relevant data interaction records, clarify the transmission details, and integrate the supplementary content into the requirement feature set.

[0059] Check the logical connection requirements. If there are unclear constraints, such as vague descriptions of the execution order constraints between certain steps, supplement the execution logic description, clarify the constraints, and integrate the supplementary content into the requirement feature set.

[0060] Step S120: Reorganize the node resource configuration of the multi-node cluster based on the demand feature set. The node resource configuration is formed by adapting and combining the processor resources, memory resources and transmission resources of each computing node in the multi-node cluster according to the demand features in the demand feature set.

[0061] Based on the generated demand feature set, the node resources of the multi-node cluster are reorganized to form a node resource configuration. This node resource configuration is formed by adapting and combining the processor resources, memory resources, and transmission resources of each computing node in the multi-node cluster according to the resource consumption trends, data dependency patterns, and logical connection requirements in the demand feature set.

[0062] For example, based on the demand for processor and memory resources in the resource consumption trend, computing nodes with corresponding resource capabilities are selected and combined; based on the requirements for transmission resources in the data dependency pattern, it is ensured that the combined node resource configuration has the transmission capability to meet the data transmission requirements.

[0063] Step S121: Collect basic resource information of each computing node in the multi-node cluster. The basic resource information includes the processor resource type, memory resource capacity and transmission resource bandwidth of each computing node in the multi-node cluster.

[0064] Basic resource information is collected for each compute node in the multi-node cluster. Processor resource type includes the processor model and number of cores used by the compute node; memory resource capacity refers to the total memory size of the compute node; transmission resource bandwidth refers to the maximum bandwidth for data transmission between the compute node and other nodes. The collected information is then organized and archived to form a basic resource information database for each compute node.

[0065] Step S122: Extract the resource types and demand scales involved in the resource consumption trends from the demand feature set, and determine the resource types that the reorganized node resource configuration needs to cover and the minimum configuration scale of each resource type.

[0066] Extract relevant information from the resource consumption trends in the demand feature set to identify the types of resources involved in the task, such as processor resources, memory resources, and transmission resources; at the same time, determine the demand scale for each type of resource, such as the computing power that processor resources need to support, the storage capacity that memory resources need to provide, and the bandwidth that transmission resources need to have.

[0067] Based on this information, determine the types of resources that must be included when reconfiguring node resources, as well as the minimum configuration size for each type of resource, to ensure that the node resource configuration can meet the basic requirements of the task.

[0068] Step S123: Perform correlation analysis between the basic resource information of each computing node and the resource requirements in the demand feature set, identify the types and scale of resources that each computing node can provide that meet the resource requirements in the demand feature set, and form a node resource adaptation list.

[0069] The resource base information of each computing node is correlated with the resource requirements in the demand feature set to determine whether the types of resources that each computing node can provide meet the requirements, and whether the scale of resources provided meets the requirements of the demand feature set.

[0070] For example, if the demand characteristics require a certain amount of processor computing power, analyze whether the processor resource types of each computing node can provide that computing power, and whether the actual computing scale they can provide meets the requirements. Compile a list of computing nodes that meet the requirements and the types and scales of resources they can provide to form a node resource adaptation list.

[0071] Step S124: Generate resource allocation rules based on the node resource adaptation list. The resource allocation rules specify the combination methods of different resource types among computing nodes. The combination methods of different resource types among computing nodes must meet the requirements of the changing trend of resource consumption in the demand characteristics.

[0072] Based on the information in the node resource adaptation list, resource allocation rules are generated. These rules specify how different resource types are combined among computing nodes, such as which computing nodes' processor resources are combined with which computing nodes' memory resources, and how transmission resources coordinate with processor and memory resources.

[0073] At the same time, these combination methods must meet the changing requirements of concentrated demand characteristics and resource consumption trends. For example, when the demand for processor resources is high in the early stage of task execution, more computing nodes with strong processor resources should be combined; when the demand for memory resources increases in the middle of the task, the combination method should be adjusted to increase the supply of memory resources.

[0074] Step S125: Select computing node resources that meet the requirements according to the resource allocation rules, and combine the processor resources, memory resources and transmission resources of the selected computing nodes according to the adaptation ratio to form the initial node resource configuration.

[0075] Based on the resource allocation rules, suitable computing node resources are selected from the node resource adaptation list. The processor, memory, and transmission resources of the selected computing nodes are then combined according to the adaptation ratio specified in the rules.

[0076] For example, if the rules require processor resources and memory resources to be combined in a set ratio, then the appropriate number and type of processor resource nodes and memory resource nodes are selected and paired together, and the corresponding transmission resources are configured to ensure data transmission. This combination forms the initial node resource configuration.

[0077] Step S1251: Extract the resource combination ratio requirements from the resource allocation rules, and determine the proportion of processor resources, memory resources and transmission resources in the node resource configuration. The proportion of processor resources, memory resources and transmission resources in the node resource configuration is set based on the resource consumption trend of the demand feature set.

[0078] Extract the requirements for resource combination ratios from the resource allocation rules, and clarify the proportion of processor resources, memory resources, and transmission resources in the node resource configuration. The setting of the above proportions refers to the resource consumption trend of the demand characteristics. For example, in the stage where the demand for processor resources is high, the proportion of processor resources in the node resource configuration is increased accordingly; in the stage where the demand for memory resources is prominent, the proportion of memory resources is increased accordingly.

[0079] By using the above methods, we can ensure that the resource combination ratio matches the resource needs of the task at different stages.

[0080] Step S1252: Traverse the node resource adaptation list, filter out computing nodes that can provide resource types that meet the resource combination ratio requirements, and form a candidate node list. The candidate node list records the resource type of each candidate node and the resource scale that each candidate node can provide.

[0081] The node resource adaptation list is traversed, and computing nodes that can provide the resource types involved in the resource combination ratio requirements are selected. For example, if the resource combination ratio requirements include specific types of processor resources, memory resources, and transmission resources, then computing nodes with these resource types are selected from the list.

[0082] The candidate node list records in detail the type of resources provided by each candidate node, such as processor resources, memory resources, or transmission resources, as well as the specific resource scale that each candidate node can provide, such as the processor's computing power, memory storage capacity, and transmission bandwidth.

[0083] Step S1253: According to the resource combination ratio requirements, select the first resource type provider node from the candidate node list and record the processor resource scale that the first resource type provider node can provide.

[0084] Following the order determined by the resource combination ratio requirements, the providing node corresponding to the first resource type is selected from the candidate node list. Assuming the first resource type is a processor resource, then computing nodes that meet the requirements of that type are selected from the candidate node list.

[0085] Record the scale of processor resources that the node can provide, including the computing power scale determined by parameters such as the number of processor cores and clock speed.

[0086] Step S1254: Based on the selected processor resource size, calculate the required matching memory resource size and the required matching transmission resource size, and select the memory resource provider node and the transmission resource provider node from the candidate node list according to the calculation results of the required matching memory resource size and the required matching transmission resource size.

[0087] Based on the scale of the selected processor resources and the required resource combination ratio, the corresponding memory and transmission resource scales are calculated. For example, if the processor resource scale is a specific computing power, the required memory storage capacity and transmission bandwidth can be calculated according to the set ratio.

[0088] Based on the calculated required memory and transmission resource scales, select memory and transmission resource providers from the candidate node list that can meet these scale requirements. Ensure that the selected nodes are compatible with the selected processor resource nodes in terms of resource scale.

[0089] Step S1255: Combine the selected processor resources, selected memory resources and selected transmission resources, and verify whether the combined resource scale meets the resource requirements of the corresponding computing stage in the demand feature set. If the combined resource scale meets the resource requirements of the corresponding computing stage in the demand feature set, a resource subconfiguration is formed.

[0090] The previously selected processor resource nodes, memory resource nodes, and transmission resource nodes are combined to form a preliminary resource combination. The scale of this combined resource is then verified to check whether it can meet the resource requirements of the corresponding computing stages in the demand feature set.

[0091] For example, if the data preprocessing stage corresponds to this resource combination, it is necessary to verify whether the combined processor computing power, memory storage capacity, and transmission bandwidth can cover the scale required by the stage in terms of resource consumption trends. If they are satisfied, then the combination is determined as a resource sub-configuration.

[0092] Step S1256: Repeat the above selection and combination operations to generate multiple resource sub-configurations for different resource consumption trend stages in the demand feature set. Each resource sub-configuration corresponds to the resource demand of a resource consumption trend stage in the demand feature set.

[0093] Following the same selection and combination process as described above, corresponding resource sub-configurations are generated for different resource consumption trend stages within the demand feature set. Since the resource consumption trends differ across stages—for example, one stage may have higher memory resource requirements while another has higher transmission resource requirements—resources need to be combined separately for each stage.

[0094] Each generated resource subconfiguration corresponds specifically to a particular resource consumption trend stage in the demand feature set, ensuring that the resource demand at each stage can be met.

[0095] Step S1257: Integrate all resource subconfigurations to enable data interaction and collaboration among the resources in the resource subconfigurations, forming an initial node resource configuration. The initial node resource configuration includes multiple resource subconfigurations and the association methods between the resource subconfigurations.

[0096] All generated resource subconfigurations are integrated. During the integration process, it is necessary to ensure that the resources of each resource subconfiguration can achieve smooth data interaction and collaborative work. For example, data processed by the previous resource subconfiguration can be successfully passed to the next resource subconfiguration through transmission resources, and the processor resources and memory resources of each subconfiguration can cooperate with each other to jointly promote task execution.

[0097] After integration, an initial node resource configuration is formed. This initial node resource configuration not only includes all resource sub-configurations, but also clarifies the association between resource sub-configurations, such as data transmission paths and collaborative working mechanisms.

[0098] Step S1258: Record the node composition of each resource sub-configuration in the initial node resource configuration, the resource scale of each resource sub-configuration in the initial node resource configuration, and the demand characteristic stage corresponding to each resource sub-configuration in the initial node resource configuration.

[0099] Each resource subconfiguration in the initial node resource configuration is recorded in detail, including which computing nodes each resource subconfiguration consists of, such as the specific identifiers of processor nodes, memory nodes, and transmission nodes; the resource scale of each resource subconfiguration, i.e., the specific numerical range of processor computing power, memory storage capacity, and transmission bandwidth; and the demand characteristic stage corresponding to each resource subconfiguration, clarifying which resource consumption trend stage the subconfiguration serves.

[0100] Step S126: Perform a test of the initial node resource configuration and record the resource utilization efficiency data of the initial node resource configuration under simulated task load. The resource utilization efficiency data reflects the degree of adaptation of the initial node resource configuration to the demand characteristics of the demand feature set.

[0101] Run tests on the initial node resource configuration to simulate the overall computing load of the large model, allowing the initial node resource configuration to process simulated data according to the set process. During the test, record resource utilization efficiency data, including processor resource utilization, memory resource utilization, transmission resource throughput, and data processing response time.

[0102] These data can reflect the degree of fit between the initial node resource configuration and the demand characteristics such as resource consumption trends and data dependency patterns. For example, high resource utilization and short response time indicate a good degree of fit.

[0103] Step S127: Based on the resource utilization efficiency data recorded during the operation test of the initial node resource configuration, adjust the resource combination ratio of the initial node resource configuration to form the final node resource configuration.

[0104] Analyzing the resource utilization efficiency data recorded during the initial node resource configuration and operation test reveals that if the utilization rate of certain resource types is too low or too high, it indicates an unreasonable resource combination ratio. For example, if the processor resource utilization rate is consistently low, while the memory resource is frequently at full load, this suggests that the processor resource allocation is too high and the memory resource allocation is insufficient.

[0105] Based on the analysis results, the combination ratio of processor resources, memory resources, and transmission resources in the initial node resource configuration is adjusted, increasing or decreasing the resource supply in the corresponding resource subconfigurations. After multiple adjustments and tests, until the resource utilization efficiency reaches an optimal state, the final node resource configuration is formed.

[0106] Step S128: Record the resource composition of the final node resource configuration, the computing nodes involved in the final node resource configuration, and the requirements characteristics of the final node resource configuration adaptation, and form a node resource configuration specification document.

[0107] Record in detail the resource composition of the final node resource configuration, including the specific types and scales of processor resources, memory resources, and transmission resources; record the identification and distribution of all computing nodes involved in the configuration; and clarify the requirements characteristics that the configuration adapts to, such as the corresponding resource consumption trend stage and data dependency mode requirements.

[0108] Step S130: Based on the resource consumption trends and logical connection requirements of the demand feature set, the overall calculation task of the large model is divided into multiple task segments, and each task segment is assigned to the corresponding node resource configuration.

[0109] By referencing the resource consumption trends of the demand characteristics set, we can understand the differences in resource requirements at different stages of the overall computational task of the large model, and use this as a basis to determine the approximate boundaries of task decomposition. Simultaneously, based on logical connection requirements, we can clarify the sequence and dependencies between each computational stage to ensure that the decomposed task segments maintain logical coherence.

[0110] The overall computational task of the large model is broken down into multiple task segments, each with specific computational content and resource requirements. Then, based on the resource requirements and logical connections of each task segment, it is allocated to the corresponding node resource configuration to ensure that the node resource configuration can meet the processing needs of the task segment and that the logical relationships between the task segments are maintained.

[0111] Step S131: Extract the time period division of resource consumption trends from the demand feature set, determine the differences in resource demand of the overall computing task of the large model in different time periods, and set the time boundary of the overall computing task of the large model based on the time period division of resource consumption trends.

[0112] Extract time-segmentation information of resource consumption trends from the demand feature set to clarify which different time periods the overall computing task of the large model is divided into, and the start and end time range of each time period. Analyze the demand for processor resources, memory resources, and transmission resources of the task within each time period to determine the differences in resource demand between different time periods, such as the high demand for processor resources in one time period and the prominent demand for memory resources in another time period.

[0113] Based on these time periods, time boundaries are set for the overall computational tasks of the large model, so that each task segment after splitting roughly corresponds to one or more time periods with similar resource demand characteristics, which facilitates subsequent resource allocation and processing.

[0114] Step S132: Based on the correspondence of computation links in the logical connection requirements, divide the computation links that can be executed independently in the overall computation task of the large model into preliminary task units, so that the logical connection between the preliminary task units meets the constraints of the requirement feature set.

[0115] Based on the description of the correspondence between computational stages in the logical connection requirements, it is determined which computational stages have a strict sequential order and which computational stages can be executed independently logically. Those computational stages that are logically independent and have relatively simple dependencies on other stages are divided into preliminary task units.

[0116] During the partitioning process, ensure that the logical connections between the initial task units conform to the constraints of the requirement feature set, such as waiting for one initial task unit to complete before another initial task unit can begin execution, to avoid logical conflicts.

[0117] Step S133: Analyze the resource requirements of each preliminary task unit, compare the resource consumption trends in the demand feature set, determine the resource consumption scale corresponding to each preliminary task unit, and adjust the preliminary task unit into a task segment of appropriate size based on the resource consumption scale corresponding to each preliminary task unit.

[0118] For each preliminary task unit, a resource requirement analysis is performed to assess its demand for processor, memory, and transmission resources during execution. The analysis results are compared with the resource consumption trends in the demand characteristic set, and the resource demand situation in the corresponding time period is referenced to determine the resource consumption scale corresponding to each preliminary task unit, such as high, medium, and low levels.

[0119] Based on the resource consumption scale of each initial task unit, adjustments are made. Excessively large initial task units are broken down into smaller parts, and excessively small initial task units are merged to form task segments of suitable size, so that their resource requirements match the carrying capacity of the subsequently allocated node resource configuration.

[0120] Step S1331: Set resource requirement analysis dimensions for each preliminary task unit. The resource requirement analysis dimensions include processor resource requirement dimension, memory resource requirement dimension and transmission resource requirement dimension. Each resource requirement analysis dimension corresponds to a type of resource consumption data in the requirement feature set.

[0121] Three resource requirement analysis dimensions are set for each initial task unit: processor resource requirement, memory resource requirement, and transmission resource requirement. The processor resource requirement dimension corresponds to data related to processor resource consumption in the requirement feature set, such as processor usage time and computing power requirements; the memory resource requirement dimension corresponds to data such as memory resource usage and storage speed requirements; and the transmission resource requirement dimension corresponds to data such as transmission resource usage frequency and bandwidth requirements.

[0122] These dimensions allow for a comprehensive analysis of the resource requirements of the initial task units.

[0123] Step S1332: Collect resource usage data for each preliminary task unit in the simulated execution environment, and record the peak values ​​of processor resource usage, memory resource usage, and transmission resource usage during the execution of each preliminary task unit.

[0124] A simulated execution environment was set up, and each preliminary task unit was run in this environment to simulate its execution process in an actual task. During the simulation, resource usage data was collected, with a focus on recording the peak processor resource usage (maximum processor resource utilization), the peak memory resource usage (maximum memory space occupied), and the peak transmission resource usage (maximum load on transmission resources during data transfer).

[0125] These peak data can reflect the maximum demand of the initial task unit for various resources.

[0126] Step S1333: Compare the peak resource usage of each resource type during the simulated execution of each preliminary task unit with the consumption trend data of the corresponding resource type in the corresponding time period of the demand feature set, analyze the matching degree of resource demand and resource consumption trend of each preliminary task unit in each resource dimension, and comprehensively determine the resource consumption scale level of each preliminary task unit.

[0127] The peak processor resource usage, peak memory resource usage, and peak transmission resource usage obtained during the simulation of each preliminary task unit are compared with the consumption trend data of the corresponding time period and resource type in the demand feature set. For example, the peak processor resource usage of a certain preliminary task unit is compared with the processor resource consumption trend data of the corresponding time period of that unit to analyze the degree of matching, such as high matching, basic matching, or no matching.

[0128] By considering the matching degree across the three resource dimensions, the resource consumption scale level of each preliminary task unit is determined. For example, units with a high matching degree and a large peak value are classified as high resource consumption scale level, while units with a relatively good matching degree but a moderate peak value are classified as medium resource consumption scale level.

[0129] Step S1334: Set the size standard of the task segment based on the resource consumption scale level of each preliminary task unit, so that the resource requirements of each task segment are within the carrying capacity of the corresponding node resource configuration.

[0130] Based on the resource consumption scale level of each preliminary task unit, set corresponding task segment size standards. For preliminary task units with high resource consumption scale levels, set smaller task segment size standards to avoid excessive resource requirements for individual task segments that exceed the carrying capacity of node resource configurations; for preliminary task units with medium and low resource consumption scale levels, relatively larger task segment size standards can be set.

[0131] By using the above methods, we can ensure that the resource requirements of each task segment are within the capacity of the corresponding node resource configuration allocated subsequently.

[0132] Step S1335: Divide the initial task units according to the size standard of the task segments to form multiple candidate task segments.

[0133] The initial task units are split according to the set task segment size standard. For initial task units with high resource consumption, they are split according to the smaller standard to obtain multiple smaller candidate task segments; for initial task units with medium and low resource consumption, they are split according to the corresponding standard to obtain a relatively small number of candidate task segments.

[0134] During the splitting process, it is necessary to ensure the integrity of each candidate task fragment, that is, to include relatively independent computational content, so as to avoid incomplete computational logic due to splitting.

[0135] Step S1336: Analyze the resource requirements of each candidate task segment, confirm that the resource consumption scale of each candidate task segment is consistent with the size standard of the task segment, and check whether the logical connection between candidate task segments meets the requirements of the requirement feature set.

[0136] Analyze the resource requirements of each candidate task segment, assessing its processor, memory, and transmission resource needs, and confirm whether its resource consumption aligns with the set task segment size standard. If inconsistencies exist, further adjustments to the candidate task segments are necessary, such as merging or splitting them again.

[0137] At the same time, check the logical connection between candidate task segments, and check whether their order and data transmission relationship meet the logical connection requirements of the requirement feature set to ensure that there are no logical contradictions or breaks.

[0138] Step S1337: Adjust the candidate task segments that do not meet the requirements, increase or decrease the size of the candidate task segments that do not meet the requirements, until the resource consumption scale of all candidate task segments meets the size standard of task segments and the logical connection between candidate task segments is normal.

[0139] For candidate task segments whose resource consumption scale does not meet the size standard or whose logical connection is problematic, adjustments are made. If the resource consumption scale is too large, the candidate task segment is further split; if it is too small, it is merged with adjacent candidate task segments that are logically related.

[0140] During the adjustment process, the scale of resource consumption and logical connection are continuously checked until all candidate task segments meet the resource consumption scale standard and the logical connection between them is normal.

[0141] Step S1338: Determine the adjusted candidate task segments as final task segments, record the resource consumption scale of each final task segment, the logical connection relationship of each final task segment, and the preliminary task unit corresponding to each final task segment, and form a task segment list.

[0142] The adjusted and qualified candidate task segments are identified as final task segments. For each final task segment, its resource consumption scale (e.g., high, medium, low) is recorded; its logical connection with other final task segments is recorded (e.g., preceding task segments, subsequent task segments); and its corresponding preliminary task unit is recorded, clarifying its origin.

[0143] The above information is compiled into a task fragment list, which displays the basic information and relationships of all final task fragments.

[0144] Step S134: Extract the resource requirement identifier and logical connection identifier for each task segment. The resource requirement identifier corresponds to the resource consumption trend in the requirement feature set, and the logical connection identifier corresponds to the logical connection requirements in the requirement feature set.

[0145] Resource requirement identifiers and logical connection identifiers are extracted from the relevant information of each task segment. Resource requirement identifiers are markers used to represent the resource requirement characteristics of the task segment, corresponding to the resource consumption trend in the requirement feature set. For example, a certain identifier corresponds to a period of high processor resource demand in the resource consumption trend. Logical connection identifiers are used to represent the logical relationship between this task segment and other task segments, corresponding to the logical connection requirements in the requirement feature set. For example, a certain identifier indicates that this task segment must be executed after another task segment.

[0146] These identifiers facilitate the subsequent association and allocation of task fragments with node resource configurations.

[0147] Step S135: Query the node resource configuration specification document to obtain the resource composition of each node resource configuration and the requirement characteristics of each node resource configuration adaptation, and establish the association between the resource requirement identifier of the task segment and the node resource configuration.

[0148] Query the node resource configuration documentation to obtain the resource composition of each node's resource configuration, such as the type and scale of processor resources, memory resources, and transmission resources; as well as the demand characteristics that each node's resource configuration adapts to, such as the corresponding resource consumption trend stage and data dependency mode.

[0149] Based on the resource requirement identifier of the task segment and the requirement characteristics of the node resource configuration adaptation, the relationship between the two is established. For example, a task segment with a certain resource requirement identifier is associated with the node resource configuration that adapts to the requirement characteristics corresponding to that identifier.

[0150] Step S136: Based on the association between the resource requirement identifier of the task segment and the node resource configuration, and combined with the logical connection requirements reflected by the logical connection identifier of the task segment, generate a task segment allocation scheme. The task segment allocation scheme includes the node resource configuration corresponding to each task segment, and ensures that task segments with logical connection relationships are allocated to node resource configurations that can realize data interaction based on their logical connection identifiers.

[0151] Based on the association between the resource requirement identifiers of task segments and the node resource configurations, and combined with the logical connection requirements reflected by the logical connection identifiers of task segments, a task segment allocation scheme is generated. This task segment allocation scheme clearly defines the node resource configurations corresponding to each task segment, ensuring that the resource requirements of the task segments can be met.

[0152] Meanwhile, for task segments with logical connections, such as those with data transfer or sequential execution relationships, they are assigned to node resource configurations that enable data interaction based on their logical connection identifiers. For example, if there is a smooth transmission channel between two node resource configurations, the collaborative execution between task segments can be guaranteed.

[0153] Step S137: The task fragment allocation scheme is sent to the scheduling unit of the multi-node cluster. The scheduling unit of the multi-node cluster transmits each task fragment to the corresponding node resource configuration according to the task fragment allocation scheme, and records the task fragment allocation record. The task fragment allocation record includes the task fragment identifier, the node resource configuration identifier, and the task fragment allocation time.

[0154] The generated task fragment allocation scheme is sent to the scheduling unit of the multi-node cluster. Upon receiving the scheme, the scheduling unit transmits each task fragment to the corresponding node resource configuration one by one, according to the scheme's specifications. During transmission, the scheduling unit must ensure the integrity and accuracy of the task fragment data to avoid data loss or corruption.

[0155] Meanwhile, the scheduling unit records the allocation of task segments, which includes a unique identifier for each task segment to distinguish different task segments; the corresponding node resource configuration identifier to specify which node resource configuration the task segment is allocated to; and the allocation time of the task segment, i.e., the specific time when the task segment is transmitted to the node resource configuration.

[0156] Step S140: Construct a data transfer channel between task fragments and node resource configurations, wherein the parameters of the data transfer channel are set based on the data dependency mode of the task fragments.

[0157] Based on the data dependency patterns of task segments, determine the data transfer requirements between each task segment, including the data source, destination, transfer volume, and transfer timing. Based on these requirements, construct data flow channels between node resource configurations for each task segment to ensure that data can flow between different node resource configurations in the expected manner.

[0158] The parameters of the data transfer channel, such as bandwidth, transmission protocol, and triggering conditions, are all set according to the specific requirements of the data dependency mode to ensure the efficiency and reliability of data transmission.

[0159] Step S141: Extract the data dependency patterns of each task segment from the set of requirements features, determine the data sources that each task segment needs to receive and the data destinations that each task segment needs to output, and form a data interaction relationship table.

[0160] Extract data dependency patterns related to each task segment from the demand feature set, and analyze in detail what data each task segment needs to receive during execution, and from which other task segments or node resource configurations this data comes; at the same time, determine the data that each task segment needs to output after execution, and to which subsequent task segments or node resource configurations this data needs to be passed.

[0161] The above information is organized into a data interaction relationship table, which shows the data input source and data output destination for each task segment.

[0162] Step S142: Based on the data source and data destination in the data interaction relationship table, determine the starting point and ending point of the data flow channel. The starting point of the data flow channel is the node resource configuration that outputs the data, and the ending point of the data flow channel is the node resource configuration that receives the data.

[0163] Based on the data sources and destinations recorded in the data interaction relationship table, determine the start and end points of each data flow channel. The start point of a data flow channel refers to the node resource configuration that outputs data; that is, the data generated after the task segment in that node resource configuration completes execution needs to be transmitted through the channel. The end point of a data flow channel refers to the node resource configuration that receives data; that is, the task segment in that node resource configuration needs to receive data from other node resource configurations before it can continue execution.

[0164] By clearly defining the starting and ending points, we can ensure that the direction of data flow is accurate.

[0165] Step S143: Analyze the amount and timing of data transmission in the data dependency pattern, determine the bandwidth parameters of the data transfer channel based on the amount of data transmission, and determine the transmission triggering conditions of the data transfer channel based on the timing of data transmission.

[0166] Analyze the data transfer volume involved in the data dependency pattern to understand the approximate data volume and frequency of each data transfer between different task segments. Based on this data transfer volume information, determine the bandwidth parameters of the data flow channel to ensure that the channel bandwidth can meet the data transfer requirements and avoid data transmission delays due to insufficient bandwidth.

[0167] Simultaneously, the timing of data transmission is analyzed, such as transmitting data immediately after the completion of a task segment, or transmitting data at a specific time point. Based on these transmission timings, the transmission trigger conditions of the data flow channel are determined. When the trigger conditions are met, the channel automatically starts data transmission, ensuring that data is transmitted at the appropriate time.

[0168] Step S144: Query the transmission resource information of the node resource configuration, determine the transmission protocol type supported by each node resource configuration, and select the channel type that matches the transmission protocol type supported by each node resource configuration. The channel type includes direct channel and relay channel.

[0169] Query the transmission resource information of the node resource configuration to understand the transmission protocol types supported by each node resource configuration, such as TCP and UDP protocols. Based on this information, select a channel type that matches the transmission protocol types supported by the start and end node resource configurations.

[0170] The main channel types include direct channels and relay channels. Direct channels are suitable for situations where the resource configurations of the starting point and the ending point nodes can be directly connected and support the same transmission protocol. Relay channels are suitable for situations where the resource configurations of the starting point and the ending point nodes cannot be directly connected, or the transmission protocols are incompatible, and data needs to be forwarded through intermediate nodes.

[0171] Step S145: Construct initial data transfer channels according to channel type, bandwidth parameters and transmission trigger conditions, assign a unique identifier to each initial data transfer channel, and record the starting point, ending point and parameter information of each initial data transfer channel.

[0172] Based on the determined channel type, bandwidth parameters, and transmission triggering conditions, the initial data flow channel is constructed. For direct connection channels, a connection is established directly between the resource configurations of the starting and ending nodes, and the corresponding bandwidth parameters and transmission triggering conditions are configured. For relay channels, a connection path is established between the resource configurations of the starting node, relay node, and ending node, and the relevant parameters and conditions are configured accordingly.

[0173] Each completed initial data transfer channel is assigned a unique identifier for channel management and identification. Simultaneously, the resource configuration identifiers of the starting and ending nodes of each initial data transfer channel, as well as channel type, bandwidth parameters, transmission trigger conditions, and other parameter information, are recorded to form a channel information archive.

[0174] Step S1451: Determine the physical connection method of the data flow channel according to the channel type. The direct connection channel adopts the method of directly connecting the node resource configurations corresponding to the starting point and the ending point of the data flow channel. The relay channel adopts the method of connecting the node resource configurations corresponding to the starting point of the data flow channel to the node resource configurations corresponding to the ending point of the data flow channel via the relay node.

[0175] Depending on the channel type, the physical connection method of the data transfer channel is determined. For direct connection channels, the physical connection method is that the node resource configuration corresponding to the starting point and the node resource configuration corresponding to the ending point are directly connected by a physical line, without the participation of other intermediate nodes, and data can be directly transmitted between the two.

[0176] For the transit channel, the physical connection method is that the node resource configuration corresponding to the starting point first establishes a connection with the transit node, and then the transit node establishes a connection with the node resource configuration corresponding to the destination, forming a connection path of starting point-transit node-destination. Data is transmitted from the starting point to the destination through the transit node.

[0177] Step S1452: For direct connection channels, query the transmission interface information of the node resource configuration corresponding to the starting point of the direct connection channel and the node resource configuration corresponding to the ending point of the direct connection channel, select the matching transmission interface, and configure the transmission protocol parameters of the matching transmission interface.

[0178] For direct connection channels, query the transmission interface information of the node resource configurations corresponding to the origin and destination, including interface type, interface rate, and supported transmission protocols. Based on this information, select a transmission interface that is supported and matched by both parties to ensure that data can be transmitted through that interface.

[0179] After selecting the transmission interface, configure the transmission protocol parameters of the interface, such as port number, data packet size, and verification method. These parameters need to be consistent with the transmission protocol requirements supported by the resource configuration of both nodes to ensure normal data transmission.

[0180] Step S1453: For the relay channel, filter the relay nodes that meet the transmission resource requirements, query the transmission interface compatibility between the relay node and the node resource configuration corresponding to the start point of the relay channel and the node resource configuration corresponding to the end point of the relay channel, configure the transmission interface parameters of the relay node, and establish the connection path from the node resource configuration corresponding to the start point of the relay channel to the relay node and the node resource configuration corresponding to the end point of the relay channel.

[0181] For relay channels, the first step is to select relay nodes that meet the transmission resource requirements. Relay nodes need to have sufficient transmission bandwidth, processing capacity, and suitable transmission interfaces to ensure the efficiency and stability of data relay.

[0182] The system checks the transmission interface compatibility between the selected relay nodes and the resource configurations of the starting and ending nodes to ensure that the transmission interfaces of the relay nodes can communicate normally with the transmission interfaces of the starting and ending node resource configurations. Then, the system configures the transmission interface parameters of the relay nodes, including the interface parameters connecting to the starting point and the interface parameters connecting to the ending point. These parameters must match the transmission interface parameters of the starting and ending points, respectively.

[0183] Finally, establish a connection path between the node resource configuration corresponding to the starting point, the relay node, and the node resource configuration corresponding to the destination to ensure that data can be transmitted smoothly along this path.

[0184] Step S1454: Adjust the upper limit of the data transfer channel's transmission rate according to the bandwidth parameters so that the transmission rate of the data transfer channel can meet the data transfer volume requirements in the data dependency mode.

[0185] Based on the previously determined bandwidth parameters, adjust the upper limit of the data transfer channel's transmission rate. The upper limit of the transmission rate refers to the maximum amount of data that the channel can transmit per unit time. By adjusting this upper limit, ensure that the transmission rate of the data transfer channel can meet the data transfer volume requirements in the data dependency mode.

[0186] For example, if a data dependency mode requires the transmission of a large amount of data within a set time, the upper limit of the channel's transmission rate needs to be increased; if the amount of data to be transmitted is small, the upper limit of the transmission rate can be appropriately reduced to avoid wasting resources.

[0187] Step S1455: Set the transmission start rules of the data flow channel based on the transmission trigger conditions. The transmission trigger conditions are that the data volume reaches a preset threshold or the time reaches a preset moment. The transmission start rules of the data flow channel include the data transmission method after the transmission trigger conditions are met.

[0188] Based on transmission trigger conditions, transmission start rules are set for the data transfer channel. When the transmission trigger condition is that the data volume reaches a preset threshold, the transmission start rule stipulates that data transmission should be started immediately when the accumulated data volume to be transmitted in the channel reaches the threshold; when the transmission trigger condition is that the time reaches a preset moment, the transmission start rule stipulates that data transmission should be started when the system time reaches the preset moment.

[0189] The transmission initiation rules also include the data transmission method after the transmission triggering conditions are met, such as transmitting all data to be transmitted at once, or transmitting data in batches, to adapt to different data transmission needs.

[0190] Step S1456: Execute the initial data transfer channel construction operation, establish the data transmission path according to the physical connection method, transmission interface parameters, bandwidth parameters and transmission start rules of the data transfer channel, and form the initial data transfer channel.

[0191] According to the determined physical connection method, transmission interface parameters, bandwidth parameters, and transmission startup rules of the data flow channel, specific construction operations are performed. For direct connection channels, the transmission interfaces of the resource configurations of the starting and ending nodes are connected according to the physical connection method, and the transmission interface parameters, bandwidth parameters, and transmission startup rules are configured. For relay channels, the transmission interfaces of the resource configurations of the starting node, relay node, and ending node are connected sequentially according to the connection path, and the corresponding parameters and rules are configured respectively.

[0192] These operations establish an actual data transmission path and form an initial data flow channel.

[0193] Step S1457: Assign a unique channel identifier to each initial data transfer channel. The unique channel identifier includes the channel type code, the starting point identifier of the initial data transfer channel, and the ending point identifier of the initial data transfer channel.

[0194] Each completed initial data flow channel is assigned a unique channel identifier, which consists of several parts, including a channel type code to distinguish whether the channel is a direct connection channel or a relay channel; a starting point identifier for the initial data flow channel, i.e., a unique identifier for the resource configuration of the starting node; and an ending point identifier for the initial data flow channel, i.e., a unique identifier for the resource configuration of the ending node.

[0195] The channel identifiers formed by the above combination method are not only unique, but also intuitively reflect the channel type and connection relationship, making it convenient for channel management and querying.

[0196] Step S1458: Establish an initial data transfer channel information record table. The initial data transfer channel information record table records the channel identifier, channel type, starting node resource configuration identifier, ending node resource configuration identifier, bandwidth parameters, transmission triggering conditions, and construction time of the initial data transfer channel.

[0197] Establish an initial data transfer channel information record table, recording relevant information for each initial data transfer channel. The record includes: channel identifier (for unique channel identification); channel type (identifying whether it's a direct connection or a relay channel); originating node resource configuration identifier and ending node resource configuration identifier (indicating the channel's connection objects); bandwidth parameters (reflecting the channel's transmission capacity); transmission trigger conditions (describing the conditions for initiating transmission); and construction time (recording the channel's completion time).

[0198] Step S146: Perform the transmission test of the initial data transfer channel, send test data from the start point to the end point of the initial data transfer channel, record the transmission time and integrity of the test data transfer, and adjust the bandwidth parameters and transmission trigger conditions of the initial data transfer channel based on the transmission time and integrity of the test data transfer.

[0199] Perform transmission tests on the initial data flow channel by sending test data from the resource configuration of the starting node to the resource configuration of the ending node. The type and size of the test data should be as close as possible to the data that needs to be transmitted in the actual task to ensure the accuracy of the test results.

[0200] During the test, the time taken for the test data transmission is recorded, that is, the time from the start of data transmission to the complete reception of data; at the same time, the integrity of the test data transmission is checked, that is, whether the data received by the receiving end is consistent with the data sent by the sending end, and whether there is any data loss or error.

[0201] Analyze the performance of the initial data transfer channel based on the time and integrity of the test data transmission. If the time is too long, it may be necessary to increase the bandwidth parameters; if there are problems with data integrity, it may be necessary to adjust the transmission triggering conditions or transmission protocol parameters. Based on the analysis results, adjust the bandwidth parameters and transmission triggering conditions of the initial data transfer channel.

[0202] Step S147: Perform a transmission test on the adjusted initial data transfer channel again. Repeat the transmission test and parameter adjustment operation of the initial data transfer channel until the data transmission time and data transmission integrity meet the requirements of the data dependency mode, thus forming the final data transfer channel.

[0203] The initial data transfer channel, after parameter adjustments, is tested again. Test data is sent using the same method as before, and transmission time and data integrity are recorded. The test results are compared with the requirements of the data dependency mode to determine whether the requirements are met.

[0204] If the requirements are still not met, continue adjusting the channel parameters and test again. Repeat the testing and adjustment process until the data transmission time is within an acceptable range and the data transmission integrity meets the requirements of the data dependency mode. At this point, the channel is determined as the final data transfer channel.

[0205] Step S148: Integrate the identifier of the final data transfer channel, the parameters of the final data transfer channel, and the task segment identifier associated with the final data transfer channel to generate a channel management list for subsequent data transfer monitoring and scheduling.

[0206] The final data transfer channel identifier, various parameters (such as bandwidth, transmission trigger conditions, channel type, etc.), and the task segment identifier associated with the channel are integrated to form a channel management list.

[0207] The channel management list records in detail the basic information and associated task segments of each final data transfer channel, which facilitates the monitoring of the channel during subsequent data transfer and timely detection and resolution of problems that occur during transmission.

[0208] Step S150: Receive the task segment processing results output by the resource configuration of each node, analyze the correlation between the task segment processing results according to the logical connection requirements of the demand feature set, and generate the final accelerated result of parallel computing of the large model.

[0209] A dedicated result receiving mechanism is deployed to receive the task fragment processing results output by each node's resource configuration in real time. During the receiving process, the results are initially verified to ensure their completeness and accuracy.

[0210] Then, based on the logical connection requirements of the demand feature set, the relationships between the processing results of each task segment are analyzed, such as which results need to be integrated in sequence and which results have data dependencies. Based on these relationships, the processing results of the task segments are integrated and processed to finally generate the final accelerated result of parallel computing of the large model.

[0211] For example, in step S151: Deploy the result receiving module to receive the task fragment processing results output by each node resource configuration in real time. Each task fragment processing result carries the corresponding task fragment identifier, the corresponding node resource configuration identifier, and the task fragment processing result generation time.

[0212] The deployment results receiving module has the ability to continuously monitor the resource configuration output of each node and can receive the processing results of task segments in real time. Each task segment processing result carries rich identification information, including the corresponding task segment identifier, which identifies which task segment the result belongs to; the corresponding node resource configuration identifier, indicating which node resource configuration output the result; and the task segment processing result generation time, that is, the specific time point when the result was generated.

[0213] Step S152: Classify the task segment processing results according to the task segment identifier, query the task segment allocation record, obtain the requirement feature information corresponding to each task segment, and associate the task segment processing results with the logical connection requirements in the requirement features.

[0214] The received task segment processing results are categorized according to the task segment identifier, and processing results belonging to the same task segment are grouped together. Then, the task segment allocation records are queried to obtain the requirement characteristic information corresponding to each task segment, such as the resource consumption trend stage and data dependency pattern corresponding to the task segment.

[0215] Associate the results of task segment processing with the logical connection requirements in the requirements characteristics, clarify the position and role of each processing result in the entire task logic chain, and the logical relationship between it and other processing results.

[0216] Step S153: Extract the logical connection requirements from the set of requirements features, sort the processed results of the classified task segments according to the logical connection requirements, and form a sorted sequence of task segment processing results. The sorted sequence of task segment processing results reflects the order of integration of task segment processing results.

[0217] Sequence rules for logical connection requirements are extracted from the set of demand features. These rules specify the execution order of each computational stage and task segment. Following these sequence rules, the processing results of the categorized task segments are sorted to determine their order in the integration process.

[0218] After sorting, a sorted sequence of task segment processing results is formed, which clearly reflects the order in which the processing results of each task segment are integrated.

[0219] Step S154: Extract the processing results of the top-ranked task segments from the sorted sequence of task segment processing results, analyze the key data contained in the top-ranked task segment processing results, and mark the key data as shared data.

[0220] The processing results of the top-ranked task segments are selected from the sorted sequence. These results are then analyzed to extract key data. Key data refers to data that significantly impacts the integration of subsequent task segment processing results, such as basic parameters and intermediate calculation results.

[0221] The aforementioned key data will be marked as shared data so that it can be used in the subsequent integration process of the relevant task fragment processing results.

[0222] Step S155: Based on the data dependency pattern and logical connection requirements, determine the subsequent task segment processing result integration stage that needs to use the shared data, and transfer the shared data to the corresponding subsequent integration stage.

[0223] Based on data dependency patterns and logical connection requirements, we analyze which subsequent task segment processing and result integration stages require the use of tagged shared data. Data dependency patterns clarify the data transfer relationships, and logical connection requirements clarify the sequence between stages. Combining these two aspects allows us to accurately determine the subsequent integration stages that require the use of shared data.

[0224] Shared data will be transmitted to these corresponding subsequent integration stages to ensure that these stages can obtain the required shared data in a timely manner when integrating the results, thus guaranteeing the smooth progress of the integration work.

[0225] Step S156: In the subsequent task segment processing result integration stage, the shared data is incorporated, the preliminary integration of the task segment processing results is performed, and the preliminary integrated data is generated.

[0226] In the subsequent task fragment processing result integration stage, the transmitted shared data will be incorporated into the task fragment processing results of this stage. Figure 2 The illustration shows exemplary hardware and software components of a large model-parallel computing acceleration system 100 for multi-node clusters, which can implement the ideas of this application, according to some embodiments of this application. For example, processor 120 can be used in the large model-parallel computing acceleration system 100 for multi-node clusters and to perform the functions in this application.

[0227] For example, a large-scale model-parallel computing acceleration system 100 applied to a multi-node cluster may include a network port 110 connected to a network, one or more processors 120 for executing program instructions, a communication bus 130, and various forms of storage media 140, such as a disk, ROM, or RAM, or any combination thereof. Exemplarily, the large-scale model-parallel computing acceleration system 100 applied to a multi-node cluster may also include program instructions stored in ROM, RAM, or other types of non-transitory storage media, or any combination thereof. The methods of this application can be implemented according to these program instructions. The large-scale model-parallel computing acceleration system 100 applied to a multi-node cluster also includes an I / O interface 150 between the computer and other input / output devices.

[0228] Furthermore, this embodiment of the invention also provides a readable storage medium, wherein computer-executable instructions are preset in the readable storage medium, and when the processor executes the computer-executable instructions, the above-mentioned method for accelerating large-scale parallel computing applied to multi-node clusters is implemented.

[0229] It should be noted that, in order to simplify the description of the present invention and thus help to understand one or more embodiments of the invention, multiple features may sometimes be grouped into one embodiment, drawing or description thereof in the foregoing description of the embodiments of the present invention.

Claims

1. A large model parallel computing acceleration method applied to a multi-node cluster, characterized in that, The method comprises: generating a demand feature set of the overall computing task of the large model, the demand feature set comprising resource consumption trends, data dependency patterns, and logical connection requirements of the overall computing task of the large model; reconfiguring node resource configurations of the multi-node cluster based on the demand feature set, the node resource configurations being formed by processor resources, memory resources, and transmission resources of each computing node in the multi-node cluster being adapted and combined according to the demand features in the demand feature set; splitting the overall computing task of the large model into multiple task segments according to the resource consumption trends and the logical connection requirements in the demand feature set, and assigning each task segment to a corresponding node resource configuration; constructing data flow channels between the node resource configurations for the task segments, the parameters of the data flow channels being set based on the data dependency patterns of the task segments; receiving task segment processing results output by each node resource configuration, analyzing the correlations between the task segment processing results according to the logical connection requirements in the demand feature set, and generating a final acceleration result of the parallel computing of the large model.

2. The large model parallel computing acceleration method for a multi-node cluster according to claim 1, characterized in that, The generation of the demand feature set of the overall computing task of the large model comprises: analyzing the execution process of the overall computing task of the large model, and extracting the processor resource occupation time, memory resource occupation, and transmission resource usage frequency of each computing link in the execution process of the overall computing task of the large model to form resource usage records of each resource type; performing time sequence analysis on the resource usage records to generate resource consumption change sequences of each computing link in the execution period of the overall computing task of the large model, and summarizing resource consumption trends of the overall computing task of the large model based on the resource consumption change sequences, wherein the resource consumption trends reflect the change law of resource demand of the overall computing task of the large model at different time periods; analyzing the data transmission relationship between each computing link, recording the path, timing, and data volume of data transmission from one computing link to another computing link, forming a data transmission map based on the record of the data transmission relationship, extracting the data dependency patterns of the overall computing task of the large model from the data transmission map, and the data dependency patterns reflecting the fixed law of data interaction between each computing link; analyzing the execution order of each computing link, determining the execution logic that must be followed between each computing link, recording the corresponding relationship between the computing link that needs to be completed in advance and the computing link that needs to be executed subsequently, forming a logical connection list, and extracting the logical connection requirements of the overall computing task of the large model from the logical connection list, wherein the logical connection requirements reflect the constraint conditions of the execution order between each computing link; integrating the resource consumption trends, the data dependency patterns, and the logical connection requirements, marking the coverage range and influence degree of the resource consumption trends, the data dependency patterns, and the logical connection requirements in the overall computing task of the large model, and forming the demand feature set; wherein, for special computing links not covered in the resource consumption trends, resource usage analysis is supplemented and the supplemented content is integrated into the demand feature set; for transmission details not determined in the data dependency patterns, data interaction records are supplemented and the supplemented content is integrated into the demand feature set; and for unclear constraint conditions in the logical connection requirements, execution logic explanations are supplemented and the supplemented content is integrated into the demand feature set.

3. The large model parallel computing acceleration method for a multi-node cluster according to claim 2, characterized in that, The resource usage records are time-series analyzed to generate resource consumption change sequences of each computing link in the execution cycle of the overall computing task of the large model, and resource consumption trends of the overall computing task of the large model are summarized based on the resource consumption change sequences, including: The resource usage records are divided into multiple time windows according to the execution time of the overall computing task of the large model, and the duration of each time window is set based on the execution rhythm of the computing link, so that complete resource usage data is contained in each time window; The processor resource occupation time, memory resource occupation amount and transmission resource usage frequency of each computing link in each time window are respectively counted to form resource statistical data of each resource type in each time window; The resource consumption change curves of each computing link are drawn with the time windows as the horizontal axis and the resource statistical data as the vertical axis, and the resource consumption change curves of each computing link reflect the changes in resource consumption in different time windows; Change nodes are extracted from the resource consumption change curves of each computing link, and the change nodes are time windows in which the resource statistical data changes obviously, and the resource data change amount corresponding to each change node is recorded; Based on the change nodes and the resource data change amount, the resource consumption change sequences of each computing link are generated, and the resource statistical data and change node information of each time window are recorded in the resource consumption change sequences of each computing link in chronological order; The resource consumption change sequences of different computing links are compared to identify computing link groups with similar change rules in the resource consumption change sequences of different computing links, and the resource consumption common characteristics of the computing links in the computing link groups are analyzed; The resource consumption common characteristics of the computing links in the computing link groups and the resource consumption unique characteristics of a single computing link are summarized, and the time distribution in the execution cycle of the overall computing task of the large model is combined to form the resource consumption trends of the overall computing task of the large model, which includes the resource types and demand scales mainly consumed by the overall computing task of the large model in different time stages; The resource consumption trends of the overall computing task of the large model are associated with the corresponding computing links and the corresponding time windows, and the influence range of the resource consumption trends of the overall computing task of the large model on the overall computing task of the large model is marked.

4. The large model parallel computing acceleration method for multi-node clusters according to claim 1, characterized in that, The node resource configuration of the multi-node cluster is reorganized based on the demand feature set, including: Resource basic information of each computing node in the multi-node cluster is collected, and the resource basic information includes processor resource types, memory resource capacities and transmission resource bandwidths of each computing node in the multi-node cluster; The resource types and demand scales involved in the resource consumption trends are extracted from the demand feature set, and the resource types and the minimum configuration scales of each resource type that need to be covered by the reorganized node resource configuration are determined; The resource basic information of each computing node and the resource demand in the demand feature set are associated and analyzed to identify the resource types and demand scales that can be provided by each computing node and meet the resource demand in the demand feature set, and a node resource adaptation list is formed; The resource allocation rule is generated based on the node resource adaptation list, and the resource allocation rule specifies a combination mode of different resource types among the computing nodes, and the combination mode of different resource types among the computing nodes needs to meet the change requirement of the resource consumption trend in the demand feature set; According to the resource allocation rule, the required computing node resources are selected, and the processor resources, memory resources and transmission resources of the selected computing nodes are combined in the adaptation proportion to form an initial node resource configuration; The running test of the initial node resource configuration is performed, and the resource utilization efficiency data of the initial node resource configuration under the simulated task load is recorded, and the resource utilization efficiency data reflects the adaptation degree of the initial node resource configuration to the demand characteristics in the demand feature set; Based on the resource utilization efficiency data recorded in the running test of the initial node resource configuration, the resource combination proportion of the initial node resource configuration is adjusted to form a final node resource configuration; The resource composition of the final node resource configuration, the computing nodes involved in the final node resource configuration and the demand characteristics adapted by the final node resource configuration are recorded to form a node resource configuration description document.

5. The large model parallel computing acceleration method for a multi-node cluster according to claim 4, characterized in that, The resource allocation rule is generated based on the node resource adaptation list, and the resource allocation rule specifies a combination mode of different resource types among the computing nodes, and the combination mode of different resource types among the computing nodes needs to meet the change requirement of the resource consumption trend in the demand feature set; The resource combination proportion requirement is extracted from the resource allocation rule to determine the proportion relationship of the processor resources, memory resources and transmission resources in the node resource configuration, and the proportion relationship of the processor resources, memory resources and transmission resources in the node resource configuration is set based on the resource consumption trend in the demand feature set; The node resource adaptation list is traversed, and the computing nodes providing the resource types meeting the resource combination proportion requirement are filtered to form a candidate node list, and the resource types of each candidate node and the resource size that each candidate node can provide are recorded in the candidate node list; According to the resource combination proportion requirement, the first resource type providing node is selected from the candidate node list, and the processor resource size that the first resource type providing node can provide is recorded; Based on the selected processor resource size, the required matched memory resource size and the required matched transmission resource size are calculated, and the memory resource providing node and the transmission resource providing node are selected from the candidate node list according to the calculation results of the required matched memory resource size and the required matched transmission resource size; The selected processor resources, the selected memory resources and the selected transmission resources are combined, and it is verified whether the combined resource size meets the resource demand of the corresponding computing link in the demand feature set, and if the combined resource size meets the resource demand of the corresponding computing link in the demand feature set, a resource sub-configuration is formed; The above selection and combination operations are repeated, and multiple resource sub-configurations are generated for different resource consumption trend stages in the demand feature set, and each resource sub-configuration corresponds to the resource demand of a resource consumption trend stage in the demand feature set; Integrate all resource sub-configurations to enable data interaction and collaboration between resources of the resource sub-configurations, and form an initial node resource configuration, the initial node resource configuration comprising a plurality of resource sub-configurations and an association manner between the resource sub-configurations; Record the node composition of each resource sub-configuration in the initial node resource configuration, the resource scale of each resource sub-configuration in the initial node resource configuration, and the corresponding demand characteristic phase of each resource sub-configuration in the initial node resource configuration.

6. The large model parallel computing acceleration method for multi-node clusters according to claim 1, characterized in that, According to the resource consumption trend and the logical connection requirement in the demand characteristic set, the overall computing task of the large model is divided into a plurality of task segments, and each task segment is assigned to a corresponding node resource configuration, including: Extracting the time period division of the resource consumption trend from the demand characteristic set, determining the resource requirement difference of the overall computing task of the large model at different time periods, and setting the time boundary of the division of the overall computing task of the large model based on the time period division of the resource consumption trend; According to the calculation link corresponding relationship in the logical connection requirement, the calculation links that can be independently executed in the overall computing task of the large model are divided into preliminary task units, so that the logical connection between the preliminary task units meets the constraint condition in the demand characteristic set; Analyzing the resource requirement of each preliminary task unit, comparing the resource consumption trend in the demand characteristic set, determining the resource consumption scale corresponding to each preliminary task unit, and adjusting the preliminary task unit to a task segment with an appropriate size based on the resource consumption scale corresponding to each preliminary task unit; Extracting the resource requirement identifier and the logical connection identifier of each task segment, the resource requirement identifier corresponding to the resource consumption trend in the demand characteristic set, and the logical connection identifier corresponding to the logical connection requirement in the demand characteristic set; Querying the node resource configuration specification document to obtain the resource composition of each node resource configuration and the demand characteristics adapted to each node resource configuration, and establishing the association relationship between the resource requirement identifier of the task segment and the node resource configuration; Based on the association relationship between the resource requirement identifier of the task segment and the node resource configuration, and combined with the logical connection requirement reflected by the logical connection identifier of the task segment, a task segment assignment scheme is generated, the task segment assignment scheme including the node resource configuration corresponding to each task segment, and ensuring that the task segments with logical connection relationship are assigned to the node resource configurations that can realize data interaction based on their logical connection identifiers; Sending the task segment assignment scheme to the scheduling unit of the multi-node cluster, and the scheduling unit of the multi-node cluster transmits each task segment to the corresponding node resource configuration according to the task segment assignment scheme, while recording the assignment record of the task segment, the assignment record of the task segment including the task segment identifier, the node resource configuration identifier, and the task segment assignment time.

7. The large model parallel computing acceleration method for a multi-node cluster according to claim 6, characterized in that, The analysis of the resource requirement of each preliminary task unit, the comparison of the resource consumption trend in the demand characteristic set, the determination of the resource consumption scale corresponding to each preliminary task unit, and the adjustment of the preliminary task unit to a task segment with an appropriate size based on the resource consumption scale corresponding to each preliminary task unit, including: setting a resource requirement analysis dimension for each preliminary task unit, the resource requirement analysis dimension including a processor resource requirement dimension, a memory resource requirement dimension, and a transmission resource requirement dimension, each resource requirement analysis dimension corresponding to a type of resource consumption data in the requirement feature set; collecting resource usage data of each preliminary task unit in a simulated execution environment, recording a peak processor resource occupation, a peak memory resource occupation, and a peak transmission resource usage during execution of each preliminary task unit; comparing the resource usage peak of each resource type in the simulated execution of each preliminary task unit with the consumption trend data of the corresponding resource type in the requirement feature set, analyzing the matching degree of the resource requirement and the resource consumption trend of each preliminary task unit in each resource dimension, and comprehensively determining the resource consumption scale level of each preliminary task unit; setting a task segment size standard based on the resource consumption scale level of each preliminary task unit, so that the resource requirement of each task segment is within the carrying range of the corresponding node resource configuration; splitting the preliminary task unit according to the task segment size standard to form multiple candidate task segments; analyzing the resource requirement of each candidate task segment, confirming that the resource consumption scale of each candidate task segment is consistent with the size standard of the task segment, and checking whether the logical connection between the candidate task segments meets the requirements in the requirement feature set; adjusting the candidate task segments that do not meet the requirements, increasing or decreasing the size of the candidate task segments that do not meet the requirements, until the resource consumption scale of all candidate task segments meets the size standard of the task segment and the logical connection between the candidate task segments is normal; determining the adjusted candidate task segments as final task segments, recording the resource consumption scale of each final task segment, the logical connection relationship of each final task segment, and the corresponding preliminary task unit of each final task segment, and forming a task segment list.

8. The large model parallel computing acceleration method for multi-node clusters according to claim 1, characterized in that, The data flow channel of the constructed task segment between the node resource configurations is parameterized based on the data dependency mode of the task segment, including: extracting the data dependency mode of each task segment from the requirement feature set, determining the data source that each task segment needs to receive and the data destination that each task segment needs to output, and forming a data interaction relationship table; determining the starting point of the data flow channel and the ending point of the data flow channel according to the data source and the data destination in the data interaction relationship table, the starting point of the data flow channel being the node resource configuration that outputs data, and the ending point of the data flow channel being the node resource configuration that receives data; analyzing the data transfer amount and data transfer timing in the data dependency mode, determining the bandwidth parameter of the data flow channel based on the data transfer amount, and determining the transmission trigger condition of the data flow channel based on the data transfer timing; querying the transmission resource information of the node resource configuration, determining the transmission protocol types supported by each node resource configuration, selecting a channel type that matches the transmission protocol types supported by each node resource configuration, and the channel type including a direct channel and a transit channel; According to the channel type, the bandwidth parameter and the transmission trigger condition, an initial data flow transfer channel is constructed, a unique identifier is allocated to each initial data flow transfer channel, and the starting point of each initial data flow transfer channel, the end point of each initial data flow transfer channel and the parameter information of each initial data flow transfer channel are recorded; A transmission test of the initial data flow transfer channel is performed, test data is sent from the starting point of the initial data flow transfer channel to the end point of the initial data flow transfer channel, the time consumption of the test data transmission and the integrity of the test data transmission are recorded, and the bandwidth parameter of the initial data flow transfer channel and the transmission trigger condition of the initial data flow transfer channel are adjusted based on the time consumption of the test data transmission and the integrity of the test data transmission; The transmission test of the adjusted initial data flow transfer channel is performed again, and the transmission test of the initial data flow transfer channel and the parameter adjustment operation of the initial data flow transfer channel are repeated until the time consumption of the data transmission and the integrity of the data transmission meet the requirements of the data dependency mode, and the final data flow transfer channel is formed; The identifier of the final data flow transfer channel, the parameter of the final data flow transfer channel and the task fragment identifier associated with the final data flow transfer channel are integrated to generate a channel management list for subsequent monitoring and scheduling of data flow transfer.

9. The large model parallel computing acceleration method for a multi-node cluster according to claim 8, characterized in that, According to the channel type, the bandwidth parameter and the transmission trigger condition, an initial data flow transfer channel is constructed, a unique identifier is allocated to each initial data flow transfer channel, and the starting point of each initial data flow transfer channel, the end point of each initial data flow transfer channel and the parameter information of each initial data flow transfer channel are recorded, including: According to the channel type, the bandwidth parameter and the transmission trigger condition, an initial data flow transfer channel is constructed, a unique identifier is allocated to each initial data flow transfer channel, and the starting point of each initial data flow transfer channel, the end point of each initial data flow transfer channel and the parameter information of each initial data flow transfer channel are recorded, including: For the direct connection channel, the transmission interface information of the node resource configuration corresponding to the starting point of the direct connection channel and the node resource configuration corresponding to the end point of the direct connection channel is queried, the matching transmission interface is selected, and the transmission protocol parameter of the matching transmission interface is configured; For the transfer channel, the transfer node meeting the transmission resource requirement is screened, the transmission interface compatibility of the transfer node, the node resource configuration corresponding to the starting point of the transfer channel and the node resource configuration corresponding to the end point of the transfer channel is queried, the transmission interface parameter of the transfer node is configured, and the connection path of the node resource configuration corresponding to the starting point of the transfer channel-transfer node-node resource configuration corresponding to the end point of the transfer channel is established; According to the bandwidth parameter, the upper limit of the transmission rate of the data flow transfer channel is adjusted, so that the transmission rate of the data flow transfer channel can meet the data transfer amount requirement in the data dependency mode; Based on the transmission trigger condition, the transmission start rule of the data flow transfer channel is set, the transmission trigger condition is that the data amount reaches a preset threshold or the time reaches a preset moment, and the data transmission mode after the transmission trigger condition is met is included in the transmission start rule of the data flow transfer channel. Performing a construction operation of an initial data flow channel, establishing a data transmission path according to a physical connection mode of the data flow channel, a transmission interface parameter, a bandwidth parameter and a transmission starting rule of the data flow channel, and forming the initial data flow channel; Allocating a unique channel identifier to each initial data flow channel, the unique channel identifier including a channel type code, a starting point identifier of the initial data flow channel and an end point identifier of the initial data flow channel; Establishing an initial data flow channel information record table, and recording the channel identifier of the initial data flow channel, the channel type of the initial data flow channel, the starting point node resource configuration identifier of the initial data flow channel, the end point node resource configuration identifier of the initial data flow channel, the bandwidth parameter of the initial data flow channel, the transmission trigger condition of the initial data flow channel and the construction time of the initial data flow channel in the initial data flow channel information record table.

10. A large model parallel computing acceleration system applied to a multi-node cluster, characterized in that, The application discloses a method for accelerating large model parallel computing of a multi-node cluster, and relates to the technical field of computer clusters. Performing a construction operation of an initial data flow channel, establishing a data transmission path according to a physical connection mode of the data flow channel, a transmission interface parameter, a bandwidth parameter and a transmission starting rule of the data flow channel, and forming the initial data flow channel; Allocating a unique channel identifier to each initial data flow channel, the unique channel identifier including a channel type code, a starting point identifier of the initial data flow channel and an end point identifier of the initial data flow channel; Establishing an initial data flow channel information record table, and recording the channel identifier of the initial data flow channel, the channel type of the initial data flow channel, the starting point node resource configuration identifier of the initial data flow channel, the end point node resource configuration identifier of the initial data flow channel, the bandwidth parameter of the initial data flow channel, the transmission trigger condition of the initial data flow channel and the construction time of the initial data flow channel in the initial data flow channel information record table. The application discloses a method for accelerating large model parallel computing of a multi-node cluster, and relates to the technical field of computer clusters. Performing a construction operation of an initial data flow channel, establishing a data transmission path according to a physical connection mode of the data flow channel, a transmission interface parameter, a bandwidth parameter and a transmission starting rule of the data flow channel, and forming the initial data flow channel; Allocating a unique channel identifier to each initial data flow channel, the unique channel identifier including a channel type code, a starting point identifier of the initial data flow channel and an end point identifier of the initial data flow channel; Establishing an initial data flow channel information record table, and recording the channel identifier of the initial data flow channel, the channel type of the initial data flow channel, the starting point node resource configuration identifier of the initial data flow channel, the end point node resource configuration identifier of the initial data flow channel, the bandwidth parameter of the initial data flow channel, the transmission trigger condition of the initial data flow channel and the construction time of the

Images