Co-inference method and device based on model dynamic segmentation and storage medium

Abstract

The application discloses a model dynamic segmentation-based collaborative reasoning method and device and a storage medium. The collaborative reasoning method is applied to a collaborative reasoning system and comprises the following steps: performing graph segmentation processing on a model calculation graph of a pre-trained model to obtain a plurality of model segments corresponding to the model calculation graph; deploying each model segment on a corresponding reasoning device of the collaborative reasoning system for model reasoning, and determining an overloaded model segment in each model segment; adjusting weight information of the overloaded model segment in the model calculation graph to obtain an adjusted model calculation graph; and performing graph segmentation processing on the adjusted model calculation graph to obtain adjusted model segments and performing deployment reasoning. The above scheme can improve the efficiency of model reasoning.

Description

Technical Field

[0001] This application relates to the field of deep learning technology, and in particular to a collaborative reasoning method, device, and storage medium based on model dynamic segmentation. Background Technology

[0002] With the rapid development of deep learning and artificial intelligence technologies, the computing power of a single inference device is far from meeting the needs of large-scale parameter models in the process of large-scale model inference calculation.

[0003] To address this challenge, distributed inference technology has emerged. Distributed inference technology enables efficient processing of large models by dividing computational tasks and / or model parameters across multiple or even thousands of computing cards for distributed processing.

[0004] However, existing model segmentation methods usually segment models based on human experience, which not only cannot adapt to dynamic load changes during model inference, resulting in low resource utilization, but also are difficult to apply to inference devices of different specifications. Summary of the Invention

[0005] This application provides at least one collaborative reasoning method, apparatus, device, and computer-readable storage medium based on model dynamic segmentation.

[0006] The first aspect of this application provides a collaborative inference method based on dynamic model segmentation, applied to a collaborative inference system, comprising: performing graph segmentation processing on the model computation graph of a pre-trained model to obtain multiple model fragments corresponding to the model computation graph; deploying each model fragment to an inference device corresponding to the collaborative inference system for model inference, and identifying overloaded model fragments in each model fragment; adjusting the weight information of the overloaded model fragments in the model computation graph to obtain an adjusted model computation graph; and performing graph segmentation processing on the adjusted model computation graph to obtain adjusted model fragments and deploying them for inference.

[0007] In one embodiment, the step of performing graph segmentation processing on the model computation graph of the pre-trained model to obtain multiple model fragments corresponding to the model computation graph includes: determining graph segmentation parameters based on the device operating capabilities of each inference device corresponding to the collaborative inference system; the graph segmentation parameters include homogeneous graph segmentation parameters and heterogeneous graph segmentation parameters; in response to the same device operating capabilities of each inference device, performing graph segmentation processing on the model computation graph based on the homogeneous graph segmentation parameters to obtain multiple model fragments corresponding to the model computation graph; in response to different device operating capabilities of each inference device, performing graph segmentation processing on the model computation graph based on the heterogeneous graph segmentation parameters to obtain multiple model fragments corresponding to the model computation graph.

[0008] In one embodiment, the isomorphic graph segmentation parameters include the number of isomorphic segments, a first load difference coefficient, and isomorphic cutting edge weight conditions. The step of performing graph segmentation processing on the model computation graph according to the isomorphic graph segmentation parameters to obtain multiple model fragments corresponding to the model computation graph includes: dividing the model computation graph into multiple model fragments with the same number of isomorphic segments, wherein the difference in operating load between each model fragment is less than or equal to the first load difference coefficient, and the sum of the edge weights between each model fragment satisfies the isomorphic cutting edge weight conditions.

[0009] In one embodiment, the heterogeneous graph segmentation parameters include a number of heterogeneous segments, a second load difference coefficient, a heterogeneous cutting edge weight condition, and a target weight array. The step of performing graph segmentation processing on the model computation graph according to the homogeneous graph segmentation parameters to obtain multiple model fragments corresponding to the model computation graph includes: dividing the model computation graph into multiple model fragments with the same number of heterogeneous segments according to each array element in the target weight array, wherein the vertex weight of each model fragment is less than or equal to the corresponding array element, the difference in runtime load between each model fragment is less than or equal to the second load difference coefficient, and the sum of the edge weights between each model fragment satisfies the heterogeneous cutting edge weight condition.

[0010] In one embodiment, the step of deploying each model fragment to the corresponding inference device of the collaborative inference system for model inference and determining the overloaded model fragment in each model fragment includes: deploying each model fragment to the corresponding inference device according to the dependency relationship between each model fragment to obtain the inference data stream of each model fragment; dividing the obtained data to be processed into multiple sub-data to be processed; inputting each sub-data to be processed into the inference data stream for model inference processing to obtain the running load of each model fragment; and determining the overloaded model fragment from each model fragment according to the running load.

[0011] In one embodiment, the operating load includes utilization. The step of inputting each piece of data to be processed into the inference data stream for model inference processing to obtain the operating load of each model shard includes: inputting each piece of data to be processed into the inference data stream for model inference processing to obtain the inference request reception rate and the inference request processing rate of each model shard; and determining the utilization rate of the model shard based on the ratio of the inference request reception rate to the inference request processing rate.

[0012] In one embodiment, the weight information includes vertex weights and edge weights. Adjusting the weight information of the overloaded model fragment in the model computation graph to obtain the adjusted model computation graph includes: obtaining the runtime load of the overloaded model fragment during model inference; adjusting the vertex weights in the overloaded model fragment according to the runtime load; obtaining the adjacent model fragments corresponding to the overloaded model fragment; and adjusting the edge weights corresponding to the overloaded model fragment according to the runtime load difference between the overloaded model fragment and the adjacent model fragments to obtain the adjusted model computation graph.

[0013] In one embodiment, the model computation graph includes vertices and edges. Before performing graph segmentation on the model computation graph of the pre-trained model to obtain multiple model slices corresponding to the model computation graph, the method further includes: acquiring the data flow between the model operators of the pre-trained model and each model operator; determining the model operators as vertices, and determining the vertex weights of the vertices based on the data processing time of the model operators; determining the data flow as edges between vertices, and determining the edge weights of each edge based on the communication time between each vertex; and constructing the model computation graph based on each vertex and its corresponding vertex weight, each edge and its corresponding edge weight.

[0014] A second aspect of this application provides a collaborative inference device based on dynamic model segmentation, comprising: a segmentation module for performing graph segmentation processing on the model computation graph of a pre-trained model to obtain multiple model fragments corresponding to the model computation graph; an inference module for deploying each model fragment to an inference device corresponding to the collaborative inference system for model inference, and determining overloaded model fragments in each model fragment; and an adjustment module for adjusting the weight information of the overloaded model fragments in the model computation graph to obtain an adjusted model computation graph; the segmentation module is further configured to perform graph segmentation processing on the adjusted model computation graph to obtain adjusted model fragments and perform deployment inference.

[0015] A third aspect of this application provides an electronic device including a memory and a processor, the processor being used to execute program instructions stored in the memory to implement the above-described collaborative reasoning method based on model dynamic segmentation.

[0016] The fourth aspect of this application provides a computer-readable storage medium having program instructions stored thereon, which, when executed by a processor, implement the above-described collaborative reasoning method based on model dynamic segmentation.

[0017] The above scheme abstracts the pre-trained model into a graph structure, obtaining the model computation graph of the pre-trained model. This graph can then be partitioned to obtain multiple model fragments. Each model fragment is deployed on its corresponding inference device in the collaborative inference system for model inference, and the load during inference is monitored to identify overloaded fragments. The weights of these overloaded fragments in the model computation graph are then adjusted to better adapt to the dynamic load changes of the actual deployment environment. The adjusted model computation graph is then further partitioned to obtain adjusted model fragments, which are then deployed and used for inference. This model fragmentation approach allows for parallel execution of the model inference process, improving inference efficiency; and the dynamic adjustment of model fragments to adapt to the actual deployment environment improves resource utilization.

[0018] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this application. Attached Figure Description

[0019] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with this application and, together with the specification, serve to explain the technical solutions of this application.

[0020] Figure 1 This is an exemplary system architecture diagram of the collaborative reasoning method based on model dynamic segmentation in this application;

[0021] Figure 2 This is a flowchart illustrating an exemplary embodiment of the collaborative reasoning method based on model dynamic segmentation of this application; Figure 3 This is a block diagram illustrating a collaborative reasoning device based on model dynamic segmentation, as shown in an exemplary embodiment of this application; Figure 4 This is a schematic diagram of the structure of an embodiment of the electronic device of this application; Figure 5 This is a schematic diagram of the structure of an embodiment of the computer-readable storage medium of this application. Detailed Implementation

[0022] The solution of this embodiment will now be described in detail with reference to the accompanying drawings.

[0023] In the following description, specific details such as particular system architectures, interfaces, and technologies are presented for illustrative purposes rather than for limiting purposes, in order to provide a thorough understanding of this application.

[0024] In this document, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the character " / " generally indicates that the preceding and following related objects have an "or" relationship. Furthermore, "many" in this document means two or more. Moreover, the term "at least one" in this document means any combination of at least two of any one or more of a plurality of objects. For example, including at least one of A, B, and C can mean including any one or more elements selected from the set consisting of A, B, and C.

[0025] For ease of understanding, one of the applicable scenarios of this application will now be illustrated by way of example.

[0026] With the rapid development of deep learning and artificial intelligence technologies, the computing power of a single inference device is far from meeting the needs of large-scale parameter models in the process of large-scale model inference calculation.

[0027] To address this challenge, distributed inference technology has emerged. Distributed inference technology enables efficient processing of large models by dividing computational tasks and / or model parameters across multiple or even thousands of computing cards for distributed processing.

[0028] However, existing model segmentation methods typically use fixed segmentation strategies devised by human experience (such as static batch processing or simple polling scheduling), which not only fail to adapt to dynamic load changes during model inference, resulting in low resource utilization, but are also difficult to apply to inference devices of different specifications.

[0029] This application proposes a collaborative reasoning method based on dynamic model segmentation, which can be applied to collaborative reasoning systems based on dynamic model segmentation. It enables real-time perception and adjustment of the inference device status and inference task load in the environment. (See reference...) Figure 1 As shown, Figure 1 This is an exemplary system architecture diagram of the collaborative reasoning method based on model dynamic segmentation in this application.

[0030] For example, the collaborative inference system can input a pre-trained model into the sharding module for static sharding, then the scheduling module deploys each model shard, and the detection module determines whether each model shard is overloaded. If so, the sharding module dynamically adjusts the running overloaded model shards, and the scheduling module can then adjust the deployment based on the adjusted model shards. The dynamic adjustment step can be continuously looped, determined by the load status of each model shard detected by the detection module.

[0031] Specifically, a collaborative reasoning system includes at least: The sharding module (or segmentation module) is mainly used for static sharding and dynamic adjustment of shards.

[0032] Detection module: Collects relevant data about the inference device and its execution of inference tasks, and determines whether the model fragments in the inference device are in an overloaded state.

[0033] Scheduling module: Based on the dependencies between model shards and the correspondence between model shards and inference devices, it performs deployment scheduling and adjustments.

[0034] Communication module: It can be set up independently or attached to the scheduling module. It uses standard protocols such as NCCL / MPI to realize data transmission between devices according to the deployment environment.

[0035] It should be noted that the collaborative reasoning system in this application is connected to each reasoning device. The reasoning devices can be set up inside the collaborative reasoning system or outside the collaborative reasoning system, and this is not limited here.

[0036] Please see Figure 2 , Figure 2 This is a flowchart illustrating an exemplary embodiment of the collaborative reasoning method based on model dynamic segmentation according to this application. Specifically, it may include the following steps: Step S110: Perform graph segmentation on the model computation graph of the pre-trained model to obtain multiple model slices corresponding to the model computation graph.

[0037] The pre-trained model can include, but is not limited to, object detection models, feature extraction models, classification models, generative models, etc. For details, please refer to the relevant technologies of model segmentation in this field, which will not be elaborated here.

[0038] The computation graph of a pre-trained model can be a graph structure abstracted from the pre-trained model. Specifically, the operators in the pre-trained model can be represented as vertices V in the graph structure (computation graph), and the data flow between operators can be represented as edges E connecting the vertices, thus obtaining the computation graph G(V, E).

[0039] Graph segmentation parameters are required during graph segmentation. These parameters guide the graph segmentation process, such as the number of segments, and may include further segmentation requirements, which will not be elaborated upon here.

[0040] It is understandable that there are multiple methods for graph segmentation processing of graph structures. Specific graph segmentation algorithms can be set as needed by referring to relevant technologies in this field, and are not limited here.

[0041] For example, this application mainly uses the METIS graph segmentation algorithm as an example for illustration. The graph segmentation parameters can be parameters set when implementing the METIS graph segmentation algorithm.

[0042] For example, the METIS graph segmentation algorithm is used to perform graph segmentation on the pre-trained model based on the graph segmentation parameters, and finally the mapping relationship between each model slice and each inference device is output.

[0043] The graph segmentation process may include, but is not limited to, splitting the pre-trained model into multiple parts (model slices) based on the segmentation results of the graph structure. This process can also be understood as performing model segmentation on the pre-trained model based on its computation graph. The model slices can then be distributed to different inference devices (such as GPUs, TPUs, CPUs, etc.) and / or servers for inference.

[0044] Depending on the granularity of the partitioning, graph partitioning can include inter-layer partitioning and / or intra-layer partitioning. For example, inter-layer partitioning divides the model layers (such as pipeline parallelism in GPT-3); intra-layer partitioning splits single-layer parameters or computation results (such as matrix partitioning in Megatron-LM); hybrid partitioning combines the above two methods (such as DeepSpeed's Zero-3 optimization).

[0045] It should be noted that a model slice can include at least one vertex, and a vertex can include at least one model operator.

[0046] Model operators refer to the operators in a neural network model, also known as operators or operator units. They are the basic building blocks that perform specific mathematical calculations or data transformations. They are similar to functions or operations, defining the rules for data flow and processing in the neural network.

[0047] For example, a convolutional neural network (CNN) based on image detection may include: a convolution operator responsible for extracting local features of the input data (such as an image); a pooling operator used to reduce the data dimensionality to reduce computational cost; and the weight summation process in a fully connected layer can also be regarded as an operator.

[0048] In general, common operator types typically include mathematical operators, activation function operators (such as ReLU and Sigmoid), normalization operators, and loss function operators. For specific details, please refer to the explanations and descriptions of related technologies in this field; they will not be elaborated upon here.

[0049] For example, before performing graph segmentation, you can choose to perform operator fusion on the pre-trained model to simplify its computational graph. Then, the computational graph can be represented as a graph structure G(V, E).

[0050] Operator fusion processing refers to the process where multiple model operators (such as matrix multiplication, addition, shifting, convolution, activation, etc.) exist in the initial pre-trained model. Multiple operators with consecutive or related operations in the model computation graph are merged into a single composite operator. This reduces the storage and data transfer of intermediate results, lowers memory usage, reduces computational latency, and thus improves overall inference speed.

[0051] Therefore, the operator in a vertex in this application can be an independent operator and / or a composite operator, without limitation here.

[0052] It should also be noted that during the construction of the model's computational graph, it is necessary to set the vertex weights for each vertex and the edge weights for each edge. The specific weight setting process can be determined based on the performance of the pre-trained model.

[0053] For example, the weight W(V) of a vertex can reflect the computational load of the operator corresponding to that vertex; the higher the computational load, the higher the vertex weight. The computational load can be reflected by the data processing time (in milliseconds) obtained during the profiling process in the actual model compilation.

[0054] For example, since different model shards are typically deployed on different inference devices, the edge weight W(E) between two model shards can be represented by the communication cost (in milliseconds) incurred from transmitting tensor data across devices (across model shards). An example of its mathematical expression is as follows:

[0055] Here, `size(activation)` represents the number of bytes occupied by the activation value tensor, indicating the amount of data communicated; `bandwidth` represents the transmission bandwidth of the communication module. If there is a large difference in bandwidth between multiple inference devices, it can be represented by taking the average (or minimum, or other statistical value). Since the tensor of the model is usually fixed after training, and the bandwidth is also available, the above edge weights can be calculated directly.

[0056] In general, the numerical ranges of vertex weights and edge weights may differ significantly. In practice, min-max normalization can be used to scale them to the same range, which will not be elaborated here.

[0057] When using the METIS graph segmentation algorithm API for graph segmentation, the topology of the model computation graph and the weight arrays of its edges and vertices can be input for processing, resulting in segmented model pieces and the corresponding inference devices for each piece. The structure of the model computation graph needs to be equivalently converted into a compressed adjacency list (CSR) for easier processing.

[0058] Therefore, model slices (different layers or blocks of the pre-trained model) can be deployed to corresponding inference devices to form an inference "pipeline". When performing inference tasks, after the data has completed the calculation of a part of the layers on one inference device, its output data (i.e., activation values) can be passed to the next inference device according to the dependency relationship so that the calculation of subsequent layers can continue.

[0059] Step S120: Deploy each model fragment on the corresponding inference device of the collaborative inference system for model inference, and determine the overloaded model fragment in each model fragment.

[0060] To illustrate the steps outlined above, after performing graph segmentation on the pre-trained model using the METIS graph segmentation algorithm based on graph segmentation parameters, vertex weights, and edge weights, the mapping relationship between each model slice and each inference device can be obtained. This essentially preliminarily determines, to a certain extent, the appropriate inference device for each model slice.

[0061] This step verifies the suitability of the current model sharding method and its deployment by examining the actual model inference process. Overloaded model sharding is considered an unsuitable method.

[0062] Therefore, each model fragment can be deployed on its corresponding inference device. After deployment, model inference is performed, and the overloaded model fragment is determined based on the operating load of each model fragment during the model inference process.

[0063] For example, each model shard is also equivalent to a subgraph in the model computation graph. When deploying the n subgraphs output by static sharding in the above example, there can be a one-to-one correspondence between the subgraphs and the inference devices. Treating each subgraph as a whole, the scheduling module can record the dependencies between each model shard to guide the data flow of the inference process, and can call the communication module to transfer and load each model shard onto the corresponding inference device for model inference.

[0064] In this application, the model inference process can refer to the pipeline parallelism (PP) method in this technical field, which divides the input data for model inference from a large batch into multiple micro-batches, and then inputs these micro-batches of data into the model inference pipeline in sequence.

[0065] This allows for overlapping and parallel operation of various model slices during model inference and computation, reducing idle time of inference devices and improving resource utilization and model inference efficiency.

[0066] Furthermore, by deploying models in shards and using parallel inference, it is possible to obtain the runtime load of each model shard (or each inference device) during the actual inference process.

[0067] Different model segments can correspond to the same or different overload judgment criteria. By comparing the operating load of each model segment with the corresponding overload judgment criteria, it can be determined whether the model segment is overloaded and the overloaded model segment in each model segment can be identified.

[0068] If there is no overload model fragmentation, the traditional model inference method can be used directly to receive data and perform inference calculations as usual, and then obtain the output inference results, which will not be elaborated here.

[0069] Step S130: Adjust the weight information of the overload model pieces in the model computation graph to obtain the adjusted model computation graph.

[0070] Based on the steps described above, after determining the overloaded model fragments in each model fragment, the weight information of the overloaded model fragments in the model computation graph (such as the vertex weights and edge weights in the previous example) can be adjusted to obtain an adjusted model computation graph. This allows for subsequent re-segmentation based on the adjusted model computation graph to obtain model fragments that are suitable for the current situation.

[0071] For example, in the graph segmentation process of this application, in order to optimize model inference efficiency and resource utilization, it is necessary to ensure that the load of a certain vertex is not too large (equivalent to the vertex weight not being too large), and also to ensure that the communication cost between each model fragment (inference device) is as small as possible (equivalent to the sum of edge weights between each vertex being as small as possible).

[0072] Therefore, if there is an overloaded model fragment, the weight information of the overloaded model fragment in the model computation graph needs to be adjusted so that it can meet more stringent inference performance requirements when it is re-segmented in the future.

[0073] In another example, in some application scenarios, to ensure graph segmentation accuracy, the adjusted model computation graph can be the entire graph of the model computation graph corresponding to the pre-trained model (referred to as the initial model computation graph here for ease of explanation). Alternatively, to reduce data processing volume and improve processing efficiency, the adjusted model computation graph can be a subgraph corresponding to each overloaded model piece in the initial model computation graph (which can be a combination of multiple subgraphs). Alternatively, it can also be a combination of the subgraphs corresponding to each overloaded model piece and the subgraphs corresponding to the model pieces adjacent to each overloaded model piece. Figure 1 The same applies to the adjusted model calculation diagram; no further restrictions are imposed here.

[0074] Step S140: Perform graph segmentation on the adjusted model computation graph to obtain adjusted model fragments and perform deployment inference.

[0075] Based on the steps described above, after obtaining the adjusted model computation graph, the graph segmentation processing method for the initial model computation graph in the previous example can be referred to. Similarly, graph segmentation processing (such as using the METIS graph segmentation algorithm) can be performed on the adjusted model computation graph to obtain the adjusted model fragments (i.e., the re-segmented model fragments) and the mapping relationship between the adjusted model fragments and the inference device.

[0076] Similarly, these adjusted model fragments are then deployed to the corresponding inference devices for inference computation.

[0077] Furthermore, after step S140, one can choose to continue executing the inference task directly with the current sharding and deployment method; or one can choose to refer to the aforementioned examples (such as the relevant examples of steps S120 and S130) and decide whether to continue the process of weight adjustment and re-sharding based on whether there are overloaded model shards in the adjusted model shards. This is not limited here.

[0078] As can be seen, this application abstracts the pre-trained model into a graph structure, obtaining the model computation graph of the pre-trained model. This graph can then be partitioned to obtain multiple model fragments. Each model fragment is deployed on the corresponding inference device of the collaborative inference system for model inference, and the load during the inference process is detected to identify overloaded model fragments. Then, the weight information of the overloaded model fragments in the model computation graph can be adjusted to make the adjusted model computation graph more adaptable to the dynamic load changes of the actual deployment environment. The adjusted model computation graph is then partitioned again to obtain adjusted model fragments, which are then deployed and used for inference. This allows the model inference process to be executed in parallel through model fragmentation, improving model inference efficiency; and dynamically adjusting the model fragments to adapt to the actual deployment environment improves resource utilization.

[0079] Based on the above embodiments, this embodiment describes the method of performing graph segmentation processing on the model computation graph of the pre-trained model in step S110 to obtain multiple model slices corresponding to the model computation graph.

[0080] It should be noted that traditional graph segmentation processing typically uses a set of predefined graph segmentation parameters to be configured and run in various application scenarios, but this is difficult to adapt to flexible and ever-changing application scenarios.

[0081] In actual implementation, this embodiment can be run by referring to the traditional method to configure graph segmentation parameters, or, in order to better adapt to different application scenarios, different graph segmentation parameters can be selected and used according to the device type of each inference device in the current application scenario during graph segmentation processing.

[0082] The device type can include homogeneous devices or heterogeneous devices. For example, homogeneous devices refer to inference devices in the current application scenario that have the same (or highly similar) construction, while heterogeneous devices refer to inference devices in the current application scenario that have different (or significantly different) constructions.

[0083] For ease of understanding and explanation, the following examples will simply use the difference in computing power (difference in device operating capability) of each inference device to represent its structural differences, and distinguish between homogeneous or heterogeneous devices. However, this application is not limited to judging homogeneous or heterogeneous devices only based on computing power in the specific implementation process, and will not be elaborated here.

[0084] For example, if multiple inference devices participating in model sharding deployment have the same computing power, these inference devices can be considered homogeneous devices; if at least one of the multiple inference devices participating in model sharding deployment has a different computing power than the other inference devices, these inference devices can be considered heterogeneous devices.

[0085] Specifically, in this embodiment, the method for performing graph segmentation processing on the model computation graph of the pre-trained model to obtain multiple model slices corresponding to the model computation graph includes at least the following steps S210 to S230: Step S210: Determine the graph segmentation parameters based on the operational capabilities of each inference device in the collaborative inference system; the graph segmentation parameters include homogeneous graph segmentation parameters and heterogeneous graph segmentation parameters.

[0086] Among them, the device operating capacity is equivalent to the operating capacity that the inference device can support when performing inference tasks (such as the computing power in the aforementioned example). The specific method for determining the device operating capacity can refer to the relevant technologies in this field.

[0087] For example, it can be determined by one or more of the following: device hardware specifications, memory bandwidth, floating point operations per second (FLOPS), and trillion basic operations per second (TOPS), which will not be elaborated here.

[0088] Step S220: In response to the fact that the device operation capabilities of each inference device are the same, the model computation graph is processed by graph segmentation according to the isomorphic graph segmentation parameters to obtain multiple model fragments corresponding to the model computation graph.

[0089] For example, if the computing power of each inference device participating in the current model sharding deployment is the same (i.e., homogeneous devices) based on the device's operating capabilities, then the homogeneous graph partitioning parameter is selected as the graph partitioning parameter, and the model computation graph is partitioned to obtain multiple model shards corresponding to the model computation graph.

[0090] Step S230: In response to the different operating capabilities of each inference device, the model computation graph is segmented according to the heterogeneous graph segmentation parameters to obtain multiple model fragments corresponding to the model computation graph.

[0091] Similarly, if the computing power of each inference device participating in the current model sharding deployment is different (i.e., heterogeneous devices) based on the device's operating capabilities, then the heterogeneous graph partitioning parameter is selected as the graph partitioning parameter to perform graph partitioning processing on the model computation graph, resulting in multiple model shards corresponding to the model computation graph.

[0092] Based on the above embodiments, this embodiment describes the method of selecting isomorphic graph segmentation parameters as graph segmentation parameters in step S220 for graph segmentation processing.

[0093] It should be noted that, in order to ensure model inference efficiency and smooth pipeline operation of inference tasks, when performing graph partitioning in application scenarios where all inference devices are homogeneous, the following conditions must be met: 1. The inference process of each model shard has a similar time consumption (to achieve load balancing). 2. The sum of edge weights across model shards is minimized or less than the edge weight threshold (to ensure low communication costs).

[0094] Therefore, the isomorphic graph segmentation parameters may include at least: the number of isomorphic segments, the first load difference coefficient, and the isomorphic cutting edge weight condition.

[0095] The isomorphic partition number nparts1 refers to the number of model fragments obtained by partitioning the model computation graph. It can be determined based on the number of inference devices, and is usually less than or equal to the number of inference devices. For example, if the number of inference devices is n, then nparts1 can be equal to n.

[0096] The first load difference coefficient, ubvec1, is a coefficient used to measure whether the inference process time of each model segment is similar (whether the load is balanced). For example, if ubvec1 = 1.05, it means that the difference in the running load between each model segment must be less than or equal to 5%. Alternatively, a load threshold can be set, and the running load of each model segment during the inference process can be compared with the load threshold (the load threshold can be preset according to empirical values ​​or determined based on the average weight of multiple model segments). The difference between the two running loads must be less than or equal to the first load difference coefficient.

[0097] The isomorphic cutting edge weight condition objtype1 refers to the condition used to control the communication cost between different model fragments (cross-model fragments).

[0098] For example, in scenarios where edge weights are not restricted, multiple graph segmentation results may be obtained. However, based on the isomorphic cutting edge weight condition, the graph segmentation result that satisfies this condition can be determined from the multiple graph segmentation results, which is the graph segmentation result with lower communication cost.

[0099] For example, referring to the relevant explanations of the METIS graph segmentation algorithm in this technical field, by setting the optimization objective in the METIS algorithm to minimize the cutting edge weight condition METIS_OBJTYPE_CUT, the graph segmentation result with the minimum communication cost can be determined from multiple graph segmentation results when performing graph segmentation processing according to the METIS algorithm.

[0100] Specifically, in this embodiment, the method for performing graph segmentation processing on the model computation graph based on isomorphic graph segmentation parameters to obtain multiple model fragments corresponding to the model computation graph includes at least the following steps: The model computation graph is divided into multiple model fragments with the same number of isomorphic partitions, and the difference in runtime load between each model fragment is less than or equal to the first load difference coefficient, and the edge weights between each model fragment satisfy the isomorphic cutting edge weight condition.

[0101] To illustrate with the aforementioned example, in scenarios where the model computation graph is segmented based on isomorphic graph segmentation parameters, the METIS graph segmentation algorithm can be used to segment the model computation graph according to the set graph segmentation parameters. The resulting model fragments can satisfy the following: the number of model fragments is the same as the number of isomorphic segments, and the difference between the running loads of each model fragment is less than or equal to the first load difference coefficient (or the difference between the running loads of each model fragment and the load threshold is less than or equal to the first load difference coefficient). Furthermore, the sum of the edge weights between each model fragment satisfies the isomorphic cutting edge weight condition (for example, each model fragment can be obtained with the minimum sum of cross-model fragment edge weights).

[0102] The runtime load of a model slice can be represented by the time consumed during inference (computation) of that model slice, and / or by the sum of the vertex weights of each vertex in the model slice, and / or by other methods; no limitation is made here. Similarly, the cross-model slice edge weights between two model slices can refer to the explanation of edge weights in the previous example, and will not be repeated here.

[0103] Based on the above embodiments, this embodiment describes the method of selecting heterogeneous graph segmentation parameters as graph segmentation parameters in step S230 for graph segmentation processing.

[0104] It should be noted that when performing graph partitioning on heterogeneous devices, it is not necessary to force a load-balanced partitioning method.

[0105] The purpose of graph segmentation based on heterogeneous graph segmentation parameters is to allocate high-weight (high-load) nodes (model fragments) to inference devices with stronger processing capabilities during the graph segmentation process, thereby achieving the effect of adaptive graph segmentation and model fragment deployment.

[0106] The heterogeneous graph segmentation parameters may include at least: heterogeneous segmentation number nparts2, second load difference coefficient ubvec2, heterogeneous cutting edge weight condition objtype2, and target weight array tpwgts.

[0107] The principles and functions of the heterogeneous segmentation number, the second load difference coefficient, and the heterogeneous cutting edge weight condition can be similarly explained in the foregoing embodiments regarding the isomorphic graph segmentation parameters. Here, different naming methods are used only to facilitate the differentiation and understanding of different examples of this application, and will not be elaborated further.

[0108] However, it should be noted that the values ​​of the segmentation parameters for homomorphic and heteromorphic graphs can be the same or different, and no restrictions are imposed here.

[0109] For example, the number of homogeneous partitions and the number of heterogeneous partitions can be the same (depending on the number of inference devices in the current application scenario). The first load difference coefficient and the second load difference coefficient can be different. For example, the first load difference coefficient (1.05) can be less than the second load difference coefficient (1.10), which can make the load of each inference device in the homogeneous device scenario more balanced than that of each inference device in the heterogeneous device scenario.

[0110] Based on this, the heterogeneous graph segmentation parameters also include a target weight array tpwgts parameter configuration, which is mainly used to represent the ideal weight ratio of each model slice. This allows the METIS algorithm to prioritize allocating high-weight model slices to inference devices with stronger computing power.

[0111] The target weight array tpwgts is an array of length nparts2. Configuring the target weight array can represent the relative proportion of the total vertex weights that each model partition is expected to achieve.

[0112] This allows for proportional deployment based on the varying computing power of different inference devices in heterogeneous device scenarios.

[0113] For example, in a heterogeneous device scenario, there are three inference devices with computing power ratios of [2.0, 1.0, 0.5]. After normalizing the computing power, it is equivalent to setting tpwgts to [2 / 3.5, 1 / 3.5, 0.5 / 3.5]. Then, the METIS algorithm can divide the model computation graph according to the ratio in this array, so that the sum of the weights of all vertices in each model slice can be less than or equal to the corresponding array element (that is, after the model slice is deployed to the corresponding inference device according to the corresponding weight to perform the inference task, it is not easy to cause overload).

[0114] Specifically, the method in this embodiment includes at least the following steps: Based on each array element in the target weight array, the model computation graph is divided into multiple model fragments with the same number of heterogeneous partitions, and the vertex weight of each model fragment is less than or equal to the corresponding array element, the difference in running load between each model fragment is less than or equal to the second load difference coefficient, and the edge weights between each model fragment satisfy the heterogeneous cutting edge weight condition.

[0115] Based on the aforementioned examples, the principle of graph segmentation based on heterogeneous graph segmentation parameters is roughly the same as that of graph segmentation based on homogeneous graph segmentation parameters. Therefore, the examples of homogeneous device scenarios can be referenced in the previous examples.

[0116] The main difference between the two scenarios is that this embodiment adds parameter configuration for the target weight array, which allows the graph segmentation process to take into account the differences in computing power between different inference devices to adapt to map segmentation and deployment.

[0117] In scenarios where the model computation graph is segmented according to heterogeneous graph segmentation parameters, the METIS graph segmentation algorithm can be used to segment the model computation graph according to the set graph segmentation parameters. The resulting model fragments can satisfy the following: the number of model fragments is the same as the number of heterogeneous segments, the vertex weight of each model fragment is less than or equal to the corresponding array element, the difference between the running loads of each model fragment is less than or equal to the second load difference coefficient (or the difference between the running load of each model fragment and the load threshold is less than or equal to the second load difference coefficient), and the sum of the edge weights between each model fragment satisfies the heterogeneous cutting edge weight condition (for example, each model fragment can be obtained with the minimum sum of cross-model fragment edge weights).

[0118] Based on the above embodiments, this embodiment describes the method in step S120 of deploying each model fragment on the inference device corresponding to the collaborative inference system for model inference and determining the overloaded model fragment in each model fragment.

[0119] Specifically, in this embodiment, each model fragment is deployed on the corresponding inference device of the collaborative inference system for model inference. The method for determining the overloaded model fragment in each model fragment includes at least the following steps S510 to S540: Step S510: Based on the dependencies between model shards, deploy each model shard to the corresponding inference device to obtain the inference data stream of each model shard.

[0120] In the process of model segmentation, the dependencies between vertices in the graph structure, or between model segments, can be determined based on the model structure of the pre-trained model. These dependencies reflect the data flow during the inference process.

[0121] Each model fragment is equivalent to a subgraph obtained by dividing the model computation graph. Each subgraph usually has a one-to-one correspondence with each inference device. Currently, each subgraph can be regarded as a whole, and the communication module transmits and loads each model fragment onto an independent inference device to form an inference data stream.

[0122] Step S520: Divide the acquired data to be processed into multiple sub-data to be processed.

[0123] Step S530: Input each sub-data to be processed into the inference data stream for model inference processing to obtain the running load of each model segment.

[0124] Based on the pipelined parallel approach, the input data (data to be processed) of the inference data stream is divided into multiple micro-batch data (sub-data to be processed) from the large batch data, and these micro-batch data are sequentially sent into the inference data stream to achieve the overlap of computation of each model slice (inference device), thereby reducing the idle time of the inference device.

[0125] During the operation of each model slice (inference device), the collaborative inference system can also obtain the corresponding operating load through communication connection.

[0126] Step S540: Determine the overloaded model fragment from each model fragment based on the operating load.

[0127] For example, an overload threshold can be preset to determine whether a model shard is overloaded. There can be one or more overload thresholds, and the overload thresholds corresponding to different inference devices can be the same or different; this is not limited here.

[0128] For example, if the operating load of a model shard is greater than the corresponding overload threshold, the model shard can be identified as an overloaded model shard. Conversely, if the operating load of a model shard is less than or equal to the corresponding overload threshold, the model shard can be identified as an unoverloaded model shard.

[0129] Based on the above embodiments, this embodiment describes the method of inputting each sub-data to be processed into the inference data stream for model inference processing in step S530 to obtain the running load of each model slice.

[0130] The operational load can also be reflected by the utilization rate of model shards. Operational load is positively correlated with utilization rate; the higher the utilization rate of a model shard, the higher its operational load is usually.

[0131] Specifically, in this embodiment, the method for inputting each sub-data to be processed into the inference data stream for model inference processing to obtain the runtime load of each model slice includes at least the following steps S610 to S620: Step S610: Input each sub-data to be processed into the inference data stream for model inference processing to obtain the inference request reception rate and inference request processing rate of each model segment.

[0132] For example, after each piece of data to be processed is input into the inference data stream for model inference processing, each model fragment in the inference data stream will start running.

[0133] The inference request reception rate refers to the number of inference requests received by a model slice per unit time (the unit can be req / s). For example, the mathematical expression for the inference request reception rate can be:

[0134] Where Δt: detection period (time window, for example, 1-10 seconds), N j This refers to the number of inference requests received by model slice j within the time window Δt.

[0135] The inference request processing rate refers to the number of inference requests that a model slice can complete per unit of time (the unit can be req / s). The mathematical expression for the inference request acceptance rate can be:

[0136] Among them, E[S j [The average time (in seconds) for processing a single inference request for model slice j.]

[0137] Step S620: Determine the utilization rate of model fragments based on the ratio of inference request reception rate to inference request processing rate.

[0138] After obtaining the inference request reception rate and the inference request processing rate, the utilization rate of the model fragments can be determined based on the ratio of the inference request reception rate to the inference request processing rate.

[0139] Utilization rate represents the proportion of time a model shard spends performing inference tasks, and is also equivalent to the ratio of the number of inference requests received by a model shard per unit to the number of push requests it can handle. Its mathematical expression can be:

[0140] For example, the overload threshold can be set to 0.9. That is, when the utilization of a model shard is greater than 0.9, the model shard is considered to be in an overloaded state, that is, the number of inference requests received by the model shard exceeds the request processing capacity of the model shard, thereby affecting the stable operation of model inference.

[0141] Based on the above embodiments, this embodiment describes the method of adjusting the weight information of the overload model pieces in the model computation graph in step S130 to obtain the adjusted model computation graph.

[0142] The weight information includes vertex weights and edge weights.

[0143] Specifically, the method in this embodiment for adjusting the weight information of overload model pieces in the model computation graph to obtain the adjusted model computation graph includes at least the following steps S710 to S740: Step S710: Obtain the runtime load of the overloaded model shard during model inference.

[0144] As shown in the example above, the utilization rate of overloaded model fragments during model inference can be obtained.

[0145] Step S720: Adjust the vertex weights in the overload model fragments according to the operating load.

[0146] Based on the utilization rate of the overloaded model fragment and the preset utilization rate coefficient, the vertex weights in the overloaded model fragment are increased so that they can be allocated to inference devices with stronger computing power in subsequent graph segmentation processes, or they can be further refined into smaller segments, which is not limited here.

[0147] Alternatively, the runtime load can also be reflected by the inference request queue length Q of the model shard.

[0148] The inference request queue length refers to the number of inference requests waiting to be processed in the input buffer of a certain model shard. If the inference request reception rate of a certain model shard exceeds the inference request processing rate, the inference request queue length of that model shard will continue to grow.

[0149] For example, in adjusting vertex weights, in addition to adjusting the original weights of vertices in the overloaded model shards based on utilization, the inference request queue length can also be considered. The mathematical expression for adjusting vertex weights can be:

[0150] in, Let i represent the original vertex weights of the vertices (operators) in the overloaded model slice, where i indicates the i-th model slice in which the vertex is located. It is the utilization rate calculated by detecting the i-th model segment. To detect the length of the inference request queue obtained by the i-th model shard, Q_i / Q_max refers to the relative load of the queue (Q_max is set to the maximum queue capacity that the inference device where the i-th model shard is located can accommodate, which can be a preset value or the maximum number of queues estimated based on the memory of the inference device, and is not limited here), α is the utilization coefficient (an empirical value of 0.3 can be taken), and β is the queue length coefficient (an empirical value of 0.2 can be taken), which can also be adjusted according to the actual deployment environment.

[0151] This allows for the adjustment of vertex weights based on the operating load (utilization and inference request queue length) of the overloaded model shards.

[0152] Step S730: Obtain the adjacent model fragments corresponding to the overloaded model fragment.

[0153] For example, adjacent model fragments can refer to model fragments that are adjacent to and connected to overloaded model fragments in the inference data stream (there can be one or more, depending on the specific scenario).

[0154] For ease of understanding, we take an adjacent model slice as an example. If the overloaded model slice is the i-th model slice in the inference data stream, then its adjacent model slice can be the (i+1)-th model slice in the inference data stream.

[0155] Step S740: Based on the difference in operating load between the overloaded model partition and its adjacent model partitions, adjust the edge weights corresponding to the overloaded model partition to obtain the adjusted model computation graph.

[0156] Based on the steps described above, for the difference in operating load (utilization difference) between overloaded model fragments and adjacent model fragments, the edge weights between overloaded model fragments and adjacent model fragments can be increased to enhance their influence on minimizing the cutting edge weight conditions in subsequent graph segmentation processes. This will not be elaborated on here.

[0157] For example, the mathematical expression for adjusting edge weights can be:

[0158] in, ρ represents the original weights of the connection edges between model partition i and model j. i ρ is the utilization rate calculated for detection of model segment i. j The utilization rate is calculated for the detection of model segment j. This is the load difference coefficient (which can be set according to actual needs, for example, 0.3). This increases the edge weights between the overloaded model fragment and its adjacent fragments.

[0159] Subsequently, the graph segmentation process described in the previous embodiments can be repeated. Similarly, the METIS algorithm can be called to perform graph segmentation on the adjusted model computation graph to obtain the adjusted model fragments (and the new mapping relationship between each model fragment and each inference device) and then deploy and infer.

[0160] The adjusted model computation graph can refer to the model computation graph composed of overloaded model pieces (i.e., a subgraph in the model computation graph of the pre-trained model), which may or may not include unoverloaded model pieces; or it can be the whole graph with the weight information adjusted using the pre-trained model, which is not limited here.

[0161] For example, after determining the overload model fragments, you can choose to extract the overload model fragments and their neighboring model fragments and merge them to obtain a computational subgraph. (This is usually a portion of the computation graph in the computation graph of the pre-trained model). The neighborhood model slice can be a k-hop neighborhood slice of the overloaded model slice (k can be greater than or equal to 1, set as needed according to the actual application scenario).

[0162] If there is an overloaded model fragment in the neighborhood model fragment, the iteration search continues using that neighborhood model fragment as the benchmark, and the final result is represented as a CSR adjacency list structure, which is to obtain the model fragment range in the specific implementation of this embodiment (which reduces the amount of computation compared to analyzing the initial model computation graph).

[0163] That is, if the adjusted model computation graph is partitioned in step S140, the METIS algorithm can be used to partition the computation subgraph. Graph segmentation involves only local adjustments, which is less costly than global adjustments.

[0164] Based on the above embodiments, this embodiment describes the method for constructing the model computation graph before step S110. The model computation graph includes vertices and edges.

[0165] Specifically, the method in this embodiment before performing graph segmentation on the model computation graph of the pre-trained model to obtain multiple model slices corresponding to the model computation graph may include at least the following steps S001 to S004: Step S001: Obtain the model operators of the pre-trained model and the data flow between each model operator.

[0166] Step S002: Determine the model operator as a vertex, and determine the vertex weight based on the data processing time of the model operator.

[0167] Step S003: Determine the data flow as edges between vertices, and determine the edge weights of each edge based on the communication time between vertices.

[0168] Step S004: Construct a model computation graph based on each vertex and its corresponding vertex weight, each edge and its corresponding edge weight.

[0169] For a detailed understanding and implementation of this embodiment, please refer to the relevant description of the model calculation graph construction process in the foregoing embodiments, which will not be repeated here.

[0170] It should be further noted that the execution entity of the model-dynamic segmentation-based collaborative reasoning method can be a model-dynamic segmentation-based collaborative reasoning device. For example, the model-dynamic segmentation-based collaborative reasoning method can be executed by a terminal device, server, or other processing device. The terminal device can be a user equipment (UE), computer, mobile device, user terminal, terminal, cellular phone, cordless phone, personal digital assistant (PDA), handheld device, computing device, in-vehicle device, wearable device, etc. In some possible implementations, this model-dynamic segmentation-based collaborative reasoning method can be implemented by the processor calling computer-readable instructions stored in memory.

[0171] Figure 3 This is a block diagram illustrating a collaborative reasoning device based on model dynamic segmentation, as shown in an exemplary embodiment of this application. Figure 3 As shown, the exemplary collaborative reasoning device 300 based on model dynamic segmentation includes: a segmentation module 310, a reasoning module 320, and an adjustment module 330. Specifically: The segmentation module 310 performs graph segmentation processing on the model computation graph of the pre-trained model to obtain multiple model slices corresponding to the model computation graph.

[0172] The inference module 320 is used to deploy each model fragment on the corresponding inference device of the collaborative inference system for model inference and to identify the overloaded model fragments in each model fragment.

[0173] The adjustment module 330 is used to adjust the weight information of the overloaded model pieces in the model computation graph to obtain the adjusted model computation graph.

[0174] The segmentation module 310 can also be used to perform graph segmentation processing on the adjusted model computation graph to obtain adjusted model fragments and perform deployment inference.

[0175] In this exemplary collaborative inference device based on dynamic model segmentation, by abstracting the pre-trained model into a graph structure, a model computation graph of the pre-trained model is obtained. This graph can then be segmented to obtain multiple model fragments. Each model fragment is deployed on the corresponding inference device of the collaborative inference system for model inference, and the load during the inference process is detected to identify overloaded model fragments. Then, the weight information of the overloaded model fragments in the model computation graph can be adjusted to make the adjusted model computation graph more adaptable to the dynamic load changes of the actual deployment environment. The adjusted model computation graph is then segmented again to obtain adjusted model fragments, which are then deployed and used for inference. This allows the model inference process to be executed in parallel through model fragmentation, improving model inference efficiency; and dynamically adjusting the model fragments to adapt to the actual deployment environment improves resource utilization.

[0176] It should be noted that the apparatus and method provided in the above embodiments belong to the same concept, and the specific ways in which each module and unit performs operations have been described in detail in the method embodiments, and will not be repeated here. In practical applications, the apparatus provided in the above embodiments can be assigned to different functional modules as needed, that is, the internal structure of the apparatus can be divided into different functional modules to complete all or part of the functions described above, and this is not a limitation.

[0177] The functions of each module can be found in the implementation examples of the collaborative reasoning method based on model dynamic segmentation, which will not be repeated here.

[0178] Please see Figure 4 , Figure 4 This is a schematic diagram of the structure of an embodiment of the electronic device of this application. The electronic device 100 includes a memory 101 and a processor 102. The processor 102 is used to execute program instructions stored in the memory 101 to implement the steps in any of the above embodiments of the collaborative reasoning method based on model dynamic segmentation. In a specific implementation scenario, the electronic device 100 may include, but is not limited to, a microcomputer or a server. In addition, the electronic device 100 may also include mobile devices such as laptops and tablets, which are not limited here.

[0179] Specifically, processor 102 controls itself and memory 101 to implement the steps in any of the above-described embodiments of the collaborative reasoning method based on model dynamic segmentation. Processor 102 may also be referred to as a CPU (Central Processing Unit). Processor 102 may be an integrated circuit chip with signal processing capabilities. Processor 102 may also be a general-purpose processor, digital signal processor (DSP), application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. A general-purpose processor may be a microprocessor or any conventional processor. Furthermore, processor 102 may be implemented using integrated circuit chips.

[0180] In this exemplary electronic device, by abstracting the pre-trained model into a graph structure, a model computation graph of the pre-trained model is obtained. This graph can then be partitioned to obtain multiple model fragments. Each model fragment is deployed on the corresponding inference device of the collaborative inference system for model inference, and the load during the inference process is detected to identify overloaded model fragments. Then, the weight information of the overloaded model fragments in the model computation graph can be adjusted to make the adjusted model computation graph more adaptable to the dynamic load changes of the actual deployment environment. The adjusted model computation graph is then partitioned again to obtain adjusted model fragments, which are then deployed and used for inference. This allows the model inference process to be executed in parallel through model fragmentation, improving model inference efficiency; and dynamically adjusting the model fragments to adapt to the actual deployment environment improves resource utilization.

[0181] Please see Figure 5 , Figure 5 This is a schematic diagram of a computer-readable storage medium according to an embodiment of the present application. The computer-readable storage medium 110 stores program instructions 111 that can be executed by a processor. The program instructions 111 are used to implement the steps in any of the above embodiments of the collaborative reasoning method based on model dynamic segmentation.

[0182] In this exemplary storage medium, by running program instructions within the storage medium, the pre-trained model is abstracted into a graph structure, resulting in the model computation graph of the pre-trained model. This graph can then be partitioned to obtain multiple model fragments. Each model fragment is deployed on the corresponding inference device of the collaborative inference system for model inference, and the load during the inference process is detected to identify overloaded model fragments. Then, the weight information of the overloaded model fragments in the model computation graph can be adjusted to make the adjusted model computation graph more adaptable to the dynamic load changes of the actual deployment environment. The adjusted model computation graph is then partitioned again to obtain adjusted model fragments, which are then deployed and used for inference. This allows the model inference process to be executed in parallel through model fragmentation, improving model inference efficiency; and dynamically adjusting the model fragments to adapt to the actual deployment environment improves resource utilization.

[0183] In some embodiments, the functions or modules of the apparatus provided in this disclosure can be used to perform the methods described in the above method embodiments. The specific implementation can be referred to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0184] The description of the various embodiments above tends to emphasize the differences between the various embodiments. The similarities or similarities between them can be referred to, and for the sake of brevity, they will not be repeated here.

[0185] In the several embodiments provided in this application, it should be understood that the disclosed methods and apparatus can be implemented in other ways. For example, the apparatus implementations described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection of devices or units may be electrical, mechanical, or other forms.

[0186] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

Claims

1. A collaborative reasoning method based on model dynamic segmentation, characterized in that, The method is applied to a collaborative reasoning system, and the method includes: The model computation graph of the pre-trained model is subjected to graph segmentation processing to obtain multiple model fragments corresponding to the model computation graph; Each model fragment is deployed to the inference device corresponding to the collaborative inference system for model inference, and the overloaded model fragment in each model fragment is identified. Adjust the weight information of the overload model pieces in the model calculation graph to obtain the adjusted model calculation graph; The adjusted model computation graph is subjected to graph segmentation processing to obtain adjusted model fragments, which are then deployed for inference.

2. The method according to claim 1, characterized in that, The graph segmentation process performed on the computation graph of the pre-trained model yields multiple model slices corresponding to the computation graph, including: The graph segmentation parameters are determined based on the operational capabilities of each inference device in the collaborative inference system; the graph segmentation parameters include isomorphic graph segmentation parameters and heteromorphic graph segmentation parameters. Since each inference device has the same operating capability, the model computation graph is segmented according to the isomorphic graph segmentation parameters to obtain multiple model fragments corresponding to the model computation graph. In response to the different operating capabilities of each inference device, the model computation graph is segmented according to the heterogeneous graph segmentation parameters to obtain multiple model fragments corresponding to the model computation graph.

3. The method according to claim 2, characterized in that, The isomorphic graph segmentation parameters include the number of isomorphic segments, a first load difference coefficient, and isomorphic cutting edge weight conditions. The step of performing graph segmentation processing on the model computation graph according to the isomorphic graph segmentation parameters to obtain multiple model pieces corresponding to the model computation graph includes: The model computation graph is divided into multiple model fragments with the same number of isomorphic partitions, and the difference in operating load between each model fragment is less than or equal to the first load difference coefficient, and the sum of the edge weights between each model fragment satisfies the isomorphic cutting edge weight condition.

4. The method according to claim 2, characterized in that, The heterogeneous graph segmentation parameters include the number of heterogeneous segments, a second load difference coefficient, heterogeneous cutting edge weight conditions, and a target weight array. The step of performing graph segmentation processing on the model computation graph according to the homogeneous graph segmentation parameters to obtain multiple model pieces corresponding to the model computation graph includes: Based on each array element in the target weight array, the model computation graph is divided into multiple model fragments with the same number of heterogeneous partitions, and the vertex weight of each model fragment is less than or equal to the corresponding array element, the difference in running load between each model fragment is less than or equal to the second load difference coefficient, and the sum of the edge weights between each model fragment satisfies the heterogeneous cutting edge weight condition.

5. The method according to claim 1, characterized in that, The step of deploying each model fragment on the corresponding inference device of the collaborative inference system for model inference, and identifying the overloaded model fragments in each model fragment, includes: Based on the dependencies between model shards, each model shard is deployed to the corresponding inference device to obtain the inference data stream of each model shard; The acquired data to be processed is divided into multiple sub-data to be processed; Each piece of data to be processed is input into the inference data stream for model inference processing to obtain the runtime load of each model slice; The overloaded model fragment is determined from each model fragment based on the operating load.

6. The method according to claim 5, characterized in that, The operational load includes utilization. The step of inputting each sub-data to be processed into the inference data stream for model inference processing to obtain the operational load of each model shard includes: Each piece of data to be processed is input into the inference data stream for model inference processing to obtain the inference request reception rate and inference request processing rate of each model fragment; The utilization rate of the model shard is determined based on the ratio of the inference request reception rate to the inference request processing rate.

7. The method according to claim 1, characterized in that, The weight information includes vertex weights and edge weights. Adjusting the weight information of the overloaded model pieces in the model computation graph to obtain the adjusted model computation graph includes: Obtain the runtime load of the overloaded model fragment during model inference; Adjust the vertex weights in the overloaded model fragments according to the operating load; Obtain the adjacent model fragments corresponding to the overloaded model fragment; Based on the difference in operating load between the overloaded model partition and the adjacent model partition, the edge weights corresponding to the overloaded model partition are adjusted to obtain the adjusted model computation graph.

8. The method according to claim 1, characterized in that, The model computation graph includes vertices and edges. Before performing graph segmentation on the pre-trained model's computation graph to obtain multiple model fragments corresponding to the computation graph, the method further includes: Obtain the data flow between the model operators of the pre-trained model and each model operator; The model operator is determined as the vertex, and the vertex weight of the vertex is determined based on the data processing time of the model operator; The data stream is defined as the edges between vertices, and the edge weights of each edge are determined based on the communication time between vertices. The model computation graph is constructed based on each vertex and its corresponding vertex weight, each edge and its corresponding edge weight.

9. An electronic device, characterized in that, The method includes a memory and a processor, the processor being configured to execute program instructions stored in the memory to implement the method according to any one of claims 1 to 8.

10. A computer-readable storage medium having program instructions stored thereon, characterized in that, When the program instructions are executed by the processor, they implement the method described in any one of claims 1 to 8.

Images