Large model reasoning method and device based on cross-domain edge device collaboration

Abstract

The invention discloses a large model reasoning method and device based on cross-domain edge device collaboration, and the method comprises the steps: dynamically sensing the available resource condition of a graphics processor in a plurality of working nodes based on a distributed hash table, and distributing a large model block to the working nodes according to the available resource condition; the available resource condition represents an unallocated working node set; receiving a user reasoning request, wherein the reasoning request at least comprises one or more reasoning tasks; based on the large model blocks allocated by the working nodes, allocating the reasoning tasks to one or more management nodes; an area where each management node is located comprises a working node set; the management node distributes the reasoning task to the working node set of the region for reasoning; the working node set comprises a plurality of working nodes. By means of the method and device, the efficiency of the distributed reasoning task can be improved, and flexible allocation of resources and high reliability of the system can be achieved.

Description

Technical Field

[0001] The present invention relates to the field of information processing, and in particular to a large model reasoning method and device based on cross-domain edge device collaboration. Background Art

[0002] At present, AI technologies such as large language models (LLMs) have shown great potential due to their powerful natural language processing and generation capabilities, especially in scenarios such as real-time conversations, intelligent customer service, and automatic translation. At the same time, 5G networks have been commercialized worldwide, with billions of mobile devices connected. High-definition real-time intelligent data services such as live video, virtual reality (VR / AR), and cloud games are becoming more and more popular in people's lives. With the increasing number of intelligent applications based on 5G networks, the Third Generation Partnership Project (3GPP) proposed the 5G-Advanced (5G-A) standard, which aims to provide general intelligent management for various 5G application scenarios, and the introduction of large model reasoning is an effective way to achieve this service.

[0003] Mobile operators have upgraded their 5G network structure to adopt a split-RAN, such as Figure 1 As shown. The central unit (CU) server of each base station manages multiple distributed unit (DU) servers, and each distributed unit is connected to multiple radio frequency units (RUs), which communicate and interact with user terminals through antennas; and the backhaul network connects the CU to the core network, so that multiple distributed base stations can be interconnected to achieve cross-domain communication. However, due to the hundreds of billions of parameters of large models (such as the commonly used open source models LLaMA and Falcon reach 405B and 180B respectively), there are three challenges in completing large model reasoning based on existing 5G base station CU / DU edge server devices.

[0004] First, currently 5G edge servers are rarely equipped with graphics processing unit (GPU) chips. There are only a few low-end dedicated chips (such as NVIDIA Quadro-E P2200 / P1000) on the market, and the computing power is far lower than the GPU chips commonly used for large model training / inference, such as A100 and H100. In addition, compared with the centralized deployment environment of multiple machines and multiple cards in data centers commonly used for large models, there is a large network delay between edge servers across base stations in different regions, which seriously affects the efficiency of collaborative reasoning. Finally, mobile operators will fail from time to time, and adding large model inference functions to the base station edge servers that are mainly responsible for communication management further increases the workload and failure risks.

[0005] There are currently some large-model inference frameworks suitable for distributed environments, typical solutions include Petals and FastDecode. Petals accelerates the inference process of large-scale models by decomposing inference tasks and executing them in parallel on multiple heterogeneous nodes. However, in weak network environments, especially in cross-regional scenarios, its data transmission delay problem is particularly obvious; at the same time, its scheduling algorithm is simple, resource allocation is unbalanced, and some nodes are prone to being idle for a long time, which affects the overall efficiency of the system. FastDecode uses a more complex resource scheduling algorithm and has a series of optimization measures to improve inference speed and accuracy, but its effect is significantly reduced in cross-regional weak network environments, because on the one hand, its resource scheduling is highly dependent on network stability and bandwidth, and on the other hand, optimization measures such as data prefetching and caching mechanisms cannot work effectively in weak networks, but instead introduce additional delays. In summary, the existing distributed large-model inference solutions are difficult to apply to large-model inference on the mobile edge side such as 5G, and cannot meet the intelligent service needs of more and more mobile applications. Summary of the invention

[0006] The present invention provides a large model reasoning method and device based on cross-domain edge device collaboration, which are used to solve the problem of low reasoning efficiency existing in the large model reasoning process in the prior art.

[0007] The present invention provides a large model reasoning method based on cross-domain edge device collaboration, the method comprising:

[0008] Based on the distributed hash table, dynamically sense the available resource conditions of the graphics processors in the multiple working nodes, and allocate the large model blocks to the working nodes according to the available resource conditions; the available resource conditions represent the unallocated working node set;

[0009] Receiving a user reasoning request, wherein the reasoning request includes at least one or more reasoning tasks;

[0010] Based on the large model blocks allocated to the working nodes, the reasoning task is allocated to one or more management nodes; each area where the management node is located includes a set of working nodes;

[0011] The management node distributes the reasoning task to the working node set in the local area for reasoning; the working node set includes multiple working nodes.

[0012] Optionally, allocating the large model blocks to the working nodes according to the available resources includes:

[0013] When the first working node set is not empty, allocating the large model blocks to the first working node set;

[0014] Calculating whether the first assigned working node set includes an idle working node;

[0015] If so, the idle working nodes are separated and a second working node set is formed.

[0016] Optionally, the method also includes: based on the real-time network delay between the working nodes across domains, selecting the optimal inference collaborative route of the inference request for inference; the optimal inference collaborative route is the optimal transmission path for the result data between the large model blocks.

[0017] Optionally, the method further includes:

[0018] Get the first time, which is the network transmission time T t , calculate the completion time T c and request waiting time T q sum;

[0019] Minimizing the first time as an optimization goal;

[0020] The working node is selected to perform reasoning based on the optimization goal.

[0021] Optionally, the method further comprises: collecting the status of the working node, and identifying whether the working node is abnormal according to the status; the abnormality comprises: resource bottleneck or failure;

[0022] If so, the execution task of the working node is migrated to other working nodes.

[0023] Optionally, the state of the working node includes: resource occupancy rate and bandwidth utilization rate, and the method includes:

[0024] Collect resource occupancy and bandwidth utilization of the working node;

[0025] When the resource occupancy rate exceeds a resource occupancy threshold or the bandwidth utilization rate exceeds a bandwidth utilization threshold, the working node is abnormal.

[0026] Optionally, the method includes: when multiple user reasoning requests are received concurrently, dynamically adjusting the batch size of the reasoning task, and performing quantization processing on the data transmitted between the working nodes.

[0027] Optionally, dynamically adjust the batch size of the inference task:

[0028] Calculating a ratio of a queue length to a batch size of the inference task;

[0029] When the ratio is greater than 1, the batch size is increased; when the ratio is less than 0.5, the batch size is decreased.

[0030] The present invention provides a large model reasoning device based on cross-domain edge device collaboration, the device comprising:

[0031] A processing unit, configured to dynamically sense available resource conditions of graphics processors in a plurality of working nodes based on a distributed hash table, and allocate large model blocks to the working nodes according to the available resource conditions; the available resource conditions represent a set of unallocated working nodes;

[0032] A receiving unit, configured to receive a user reasoning request, wherein the reasoning request includes at least one or more reasoning tasks;

[0033] A first allocation unit is used to allocate the reasoning task to one or more management nodes based on the large model blocks allocated to the working nodes; each area where the management node is located includes a set of working nodes;

[0034] The second allocation unit is used for the management node to allocate the reasoning task to the working node set in the local area for reasoning; the working node set includes multiple working nodes.

[0035] A computer-readable storage medium of the present invention stores one or more programs, and the one or more programs can be executed by one or more processors to implement any of the methods described above.

[0036] The method and device of the present invention can effectively utilize low-end edge devices (such as low-power 5G edge servers) across geographically distributed areas to realize large-model reasoning. It can not only improve the efficiency of distributed reasoning tasks, but also realize flexible allocation of resources and high reliability of the system. It is also suitable for other related scenarios such as collaborative reasoning using idle devices by multiple organizations such as companies and universities. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 It is a schematic diagram of 5G network architecture in the prior art;

[0038] Figure 2 This is a flow chart of a large model reasoning method based on cross-domain edge device collaboration in an embodiment of the present invention;

[0039] Figure 3 is a schematic diagram of an allocation strategy for a set of working nodes in an embodiment of the present invention;

[0040] Figure 4 It is a schematic diagram of the structure of a large model inference device based on cross-domain edge device collaboration in an embodiment of the present invention. DETAILED DESCRIPTION

[0041] The present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the present invention, rather than to limit the present invention. It should also be noted that, for ease of description, only parts related to the present invention, rather than all structures, are shown in the accompanying drawings.

[0042] The specific embodiment of the present invention provides a large model reasoning method based on cross-domain edge device collaboration, whose core architecture consists of two types of nodes: a group of management nodes and a set of working nodes in the corresponding area, which correspond to CU and DU servers respectively in the 5G scenario. Each area has a management node to coordinate the working node set in the area to perform reasoning tasks. The method is specifically as follows Figure 2 As shown, including:

[0043] Step 100, based on the distributed hash table, dynamically sense the available resource conditions of the graphics processors in multiple working nodes, and allocate large model blocks to the working nodes according to the available resource conditions; the available resource conditions represent the set of unallocated working nodes. The management node will detect the available resource conditions of the graphics processors in each working node through the distributed hash table (DHT), determine the resource load according to the available resource conditions, and then allocate the large model blocks to the working nodes according to the resource load to dynamically adjust the task allocation to ensure the best utilization of computing resources in the region. Among them, the distributed hash table (DHT) can realize the monitoring and coordination of the status of decentralized working nodes across regions, dynamically sense the storage capacity and other resource conditions of the graphics processor GPU, and reasonably allocate large model blocks to maximize the GPU resource utilization of all working nodes.

[0044] Step 200, receiving a user reasoning request, wherein the reasoning request includes at least one or more reasoning tasks. Specifically, the user submits the user reasoning request to the master node through the AI ​​interface, that is, the master node receives the user reasoning request. Among them, the management node will select the management node with the smallest round-trip delay (RTT) among other management nodes as the master node.

[0045] Step 300, based on the large model blocks allocated to the working nodes, the reasoning tasks are allocated to one or more management nodes; each region where the management node is located includes a set of working nodes. In this step, it is equivalent to the master node allocating corresponding reasoning tasks to other management nodes according to the available resources in each region.

[0046] Step 400: the management node distributes the reasoning task to the working node set in the region for reasoning; the working node set includes multiple working nodes. Based on step 300, the management node further distributes the reasoning task to achieve coordination of the reasoning task. After the reasoning is completed, the result is returned to the user via the AI ​​interface.

[0047] The method described in the above embodiment of the present invention can effectively utilize low-end edge devices (such as low-power 5G edge servers) across geographically distributed areas to realize large model reasoning. With the organic combination of AI interface, master node coordination, management node task allocation, efficient computing of working nodes, and DHT cross-region monitoring, it can not only improve the efficiency of distributed reasoning tasks, but also realize flexible allocation of resources and high reliability of the system.

[0048] The large model inference method based on cross-domain edge device collaboration described in the specific embodiment of the present invention preferably allocates the large model blocks to the working nodes according to the available resources, including:

[0049] When the first working node set is not empty, allocating the large model blocks to the first working node set;

[0050] Calculating whether the first assigned working node set includes an idle working node;

[0051] If so, the idle working nodes are separated and a second working node set is formed.

[0052] Specifically, first set the pointer p to point to the initial position of the large model block L;

[0053] When the first set of working nodes is not empty, repeatedly perform the following operations:

[0054] The large model blocks are allocated to the first working node set. Specifically, they can be allocated to the first working node and the most suitable sub-node set in it.

[0055] Calculate whether the first working node set also includes idle working nodes, that is, whether there are still idle parts in the first working node set after excluding the child node set. Specifically, the nodes exceeding the predefined threshold coefficient a (the empirical value is 0.3)*total node capacity are regarded as idle working nodes.

[0056] If there are idle working nodes, the idle working nodes are separated and formed into a second working node set, and the second working node set is the unallocated nodes.

[0057] Then update the large model block and the second working node, and point the pointer p to the initial position of the updated large model block.

[0058] Returns the set of allocated worker nodes.

[0059] like Figure 3As shown, it includes work node sets A, B, C, and D, among which work node sets A and part of B can support a complete large model reasoning. Work node sets C, D, and B' (a new work node set consisting of the remaining capacity after work node set B is allocated) can also support a complete large model reasoning. Preferably, multiple copies are generated in the above allocation process, but since the location of user requests is unpredictable, storing multiple copies helps to select the shortest reasoning path requested by the user, thereby providing the best reasoning service.

[0060] In a preferred embodiment, by abstracting the large model into a computational task graph, each worker node is ensured to know which layers of the large model it is responsible for processing. For example, once worker node A is designated as the starting point for user reasoning, the next worker node in the sequence can be naturally determined to ensure that the reasoning task is completed.

[0061] In a preferred embodiment of the present invention, pipeline parallelism is used between multiple working nodes to accelerate task completion, and tensor parallelism is used within a single working node to decompose the model so that the calculation can be performed in parallel on multiple GPUs. This mechanism can reduce the idle time (bubble time) in the reasoning process and further improve the resource utilization of the above reasoning process.

[0062] The large model reasoning method based on cross-domain edge device collaboration described in the specific embodiment of the present invention is preferably further comprised of: based on the real-time network delay between the working nodes across the domain, selecting the optimal reasoning collaborative route of the reasoning request to perform reasoning to optimize the overall time of large model reasoning on the cross-domain low-end edge device (such as 5G edge server); the optimal reasoning collaborative route is the optimal transmission path of the result data between the large model blocks. When facing multi-user reasoning requests, by accurately estimating the computing time of each working node, adjusting task allocation at peak times to reduce congestion, reducing queuing delays under high concurrency, and minimizing the overall processing time of the reasoning task.

[0063] The large model reasoning method based on cross-domain edge device collaboration described in the specific embodiment of the present invention is preferably further comprising:

[0064] Get the first time, which is the network transmission time T t , calculate the completion time T c and request waiting time T q The sum of T t +T c +T q ;

[0065] Minimize the first time as the optimization target; that is, the optimization target is min{T t +T c +T q}.

[0066] The working node is selected to perform reasoning based on the optimization goal.

[0067] Specifically, in terms of time acquisition, the calculation completion time T of each area is updated regularly through the recorded data c The management node obtains the network address of each working node in the region and measures the delay, pre-calculates the network transmission time between regions, and obtains the total network transmission time T by adding the transmission delay of the user request to the start and end nodes of the route. t According to the current request queue length, estimate the request waiting time T of the user reasoning request q .

[0068] The design of this optimization goal can prevent the occurrence of computational overload as much as possible. When the number of user reasoning requests for a certain working node is too large, since the optimization goal balances the network transmission time and the request waiting time, the next user reasoning request that appears near the working node will choose a farther working node to complete the task. This avoids the phenomenon of a single working node being overloaded. In addition, since a trade-off is made between the three times, it ensures that the system can still respond quickly to user reasoning requests under high load. Experimental results show that when the number of concurrent requests increases, the reasoning delay using the random routing selection scheme increases significantly, while the reasoning delay using the method of the embodiment of the present invention increases relatively slowly, and the GPU load remains relatively balanced between various areas, avoiding waste of resources.

[0069] The large model reasoning method based on cross-domain edge device collaboration described in the specific embodiment of the present invention is preferably further comprising: collecting the state of the working node, and identifying whether the working node is abnormal according to the state; the abnormality includes: resource bottleneck or failure;

[0070] If so, the execution task of the working node is migrated to other working nodes.

[0071] Specifically, in the embodiment of the present invention, when a working node fails or a resource bottleneck occurs, fault detection is achieved by monitoring the status of the working node. The system automatically switches to the backup node and reallocates tasks to ensure the continuity of the reasoning process. When a backup node is missing, the required large model blocks are dynamically loaded to the load-balanced working node, and the reasoning path is rebuilt to ensure that task execution is not interrupted.

[0072] The large model reasoning method based on cross-domain edge device collaboration described in the specific embodiment of the present invention is preferably that the status of the working node includes: resource occupancy rate and bandwidth utilization rate, and the method includes:

[0073] Collect resource occupancy and bandwidth utilization of the working node;

[0074] When the resource occupancy rate exceeds a resource occupancy threshold or the bandwidth utilization rate exceeds a bandwidth utilization threshold, the working node is abnormal.

[0075] The management node periodically collects the GPU and video memory usage, network bandwidth, and inference task execution status of each computing node, and compares them with the set thresholds to determine whether the fault handling mechanism needs to be started.

[0076] Specifically, the management node continuously monitors the status of each distributed working node, including resource utilization, bandwidth utilization, and computing task execution progress, to ensure that each working node operates within an acceptable resource usage range. To achieve status monitoring, a resource occupancy threshold T is defined. res and bandwidth utilization threshold T bw , when the resource occupancy rate Res of the working node is detected i ≥T res Or bandwidth utilization Bw i ≤T bw When the resource usage rate is higher than the resource usage threshold or a resource anomaly occurs, the management node automatically marks the working node as a faulty node. To reduce unnecessary migration of inference tasks, multiple consecutive sampling verifications are performed before the inference task is transferred to avoid frequent transfer of inference tasks due to short-term fluctuations.

[0077] When a work node is determined to be abnormal, the inference task is reallocated to other available work nodes. During the inference task reallocation process, work nodes with smaller current inference task loads are given priority to avoid resource waste and improve response speed. Assuming that there are currently n optional work nodes, by calculating the load rate Load of each work node j Select the optimal allocation node, the calculation formula is: Load j =Task j / Cap j Among them, Task j is the current number of tasks of the jth worker node, Cap j The processing capacity of the working node. Prioritize tasks to Load j Minimum number of working nodes.

[0078] In the absence of available worker nodes, the missing large model blocks will be automatically and dynamically loaded on the load-balanced worker nodes to ensure the execution of the inference task. To this end, the size of the loaded large model block is calculated based on the remaining memory of the worker node. k , the formula is: Block k ≤Mem j -T mem Among them, Mem j is the remaining memory of worker node j, T mem Reserve a threshold for memory. During dynamic loading, prioritize the worker nodes with the largest remaining memory for loading large models in blocks, thereby achieving efficient resource utilization and fault recovery. Ensure stable operation when any worker node fails or a new worker node is needed. Ensure the continuity of reasoning tasks and data consistency through real-time monitoring, task redistribution, and dynamic loading.

[0079] The large-model inference method based on cross-domain edge device collaboration described in a specific embodiment of the present invention preferably includes: when multiple user inference requests are received concurrently, dynamically adjusting the batch size of the inference task to improve resource utilization and system response speed; and quantizing the data transmitted between the working nodes to reduce the accuracy of the transmitted data to reduce the communication volume across working nodes and improve the inference efficiency, thereby optimizing the system response speed and resource utilization efficiency in a high concurrent request environment and ensuring the stability of the inference task.

[0080] The large model reasoning method based on cross-domain edge device collaboration described in the specific embodiment of the present invention preferably dynamically adjusts the batch size of the reasoning task:

[0081] Calculating a ratio of a queue length to a batch size of the inference task;

[0082] When the ratio is greater than 1, the batch size is increased; when the ratio is less than 0.5, the batch size is decreased.

[0083] Specifically, in order to adapt to changes in the queue, a dynamic batch management mechanism is set up on the inference task queue to balance the load pressure by adjusting the batch size in real time. i To reflect the relationship between queue length and batch size, the calculation formula is: Queue i =Queue_Length j / Batch_Size j When Queue i >1, increase the batch size to speed up task processing; when the Queue iWhen <0.5, the batch size will be reduced to reduce resource usage and improve response speed. The dynamic batch processing mechanism can significantly improve system performance in high-concurrency scenarios.

[0084] Preferably, under high load conditions, the load-aware communication quantization mechanism is triggered to reduce the amount of data transmitted across nodes to optimize communication efficiency. Specifically, the data is quantized during transmission between working nodes, and high-precision data is segmented and quantized into smaller data blocks to save bandwidth. The communication quantization process is defined as: Quantized_Data = Qriginal_Data / Quantization_Factor. The Quantization_Factor is dynamically adjusted according to the real-time load. When the communication delay between working nodes exceeds the threshold T latency When the quantization mechanism is automatically started and quantization is applied between nodes to improve communication efficiency while ensuring the inference accuracy.

[0085] The specific embodiment of the present invention also provides a large model reasoning device based on cross-domain edge device collaboration, such as Figure 4 As shown, the device comprises:

[0086] The processing unit 401 is used to dynamically sense the available resource conditions of the graphics processors in the multiple working nodes based on the distributed hash table, and allocate the large model blocks to the working nodes according to the available resource conditions; the available resource conditions represent the unallocated working node set;

[0087] A receiving unit 402 is configured to receive a user reasoning request, wherein the reasoning request includes at least one or more reasoning tasks;

[0088] A first allocating unit 403 is used to allocate the reasoning task to one or more management nodes based on the large model blocks allocated to the working nodes; each area where the management node is located includes a working node set;

[0089] The second allocating unit 404 is used for the management node to allocate the reasoning task to the working node set in the local area for reasoning; the working node set includes multiple working nodes.

[0090] An embodiment of the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement any of the methods described above.

[0091] The method and device described in the embodiment of the present invention can effectively utilize low-end edge devices (such as low-power 5G edge servers) across geographical distribution areas to realize large model reasoning. With the organic combination of AI interface, master node coordination, management node task allocation, efficient computing of working nodes, and DHT cross-region monitoring, it can not only improve the efficiency of distributed reasoning tasks, but also realize flexible allocation of resources and high reliability of the system, showing reasoning performance far exceeding the existing distributed large model reasoning framework.

[0092] It should be understood that in the various embodiments of this document, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this document.

[0093] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0094] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0095] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0096] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0097] The foregoing description of specific exemplary embodiments of the present invention is for the purpose of illustration and demonstration. These descriptions are not intended to limit the present invention to the precise form disclosed, and it is clear that many changes and variations can be made based on the above teachings. The purpose of selecting and describing the exemplary embodiments is to explain the specific principles of the present invention and its practical application, so that those skilled in the art can realize and utilize various different exemplary embodiments of the present invention and various different selections and changes. The scope of the present invention is intended to be limited by the claims and their equivalents.

Claims

1. A large model reasoning method based on cross-domain edge device collaboration, characterized in that: The method comprises: Based on the distributed hash table, dynamically sense the available resource conditions of the graphics processors in the multiple working nodes, and allocate the large model blocks to the working nodes according to the available resource conditions; the available resource conditions represent the unallocated working node set; Receiving a user reasoning request, wherein the reasoning request includes at least one or more reasoning tasks; Based on the large model blocks allocated to the working nodes, the reasoning task is allocated to one or more management nodes; each area where the management node is located includes a set of working nodes; The management node distributes the reasoning task to the working node set in the local area for reasoning; the working node set includes multiple working nodes.

2. The large model reasoning method based on cross-domain edge device collaboration according to claim 1 is characterized in that: Allocating the large model blocks to the working nodes according to the available resources includes: When the first working node set is not empty, allocating the large model blocks to the first working node set; Calculating whether the first assigned working node set includes an idle working node; If so, the idle working nodes are separated and a second working node set is formed.

3. The large model reasoning method based on cross-domain edge device collaboration according to claim 1 is characterized in that: The method also includes: based on the real-time network delay between the working nodes across domains, selecting the optimal inference collaborative route of the inference request for inference; the optimal inference collaborative route is the optimal transmission path for the result data between the large model blocks.

4. The large model reasoning method based on cross-domain edge device collaboration according to claim 3 is characterized in that: The method further comprises: Get the first time, which is the network transmission time T t , calculate the completion time T c and request waiting time T q sum; Minimizing the first time as an optimization goal; The working node is selected to perform reasoning based on the optimization goal.

5. The large model reasoning method based on cross-domain edge device collaboration according to claim 1 is characterized in that: The method further comprises: collecting the status of the working node, and identifying whether the working node is abnormal according to the status; the abnormality comprises: resource bottleneck or failure; If so, the execution task of the working node is migrated to other working nodes.

6. The large model reasoning method based on cross-domain edge device collaboration according to claim 5 is characterized in that: The state of the working node includes: resource occupancy rate and bandwidth utilization rate. The method includes: Collect resource occupancy and bandwidth utilization of the working node; When the resource occupancy rate exceeds a resource occupancy threshold or the bandwidth utilization rate exceeds a bandwidth utilization threshold, the working node is abnormal.

7. The large model reasoning method based on cross-domain edge device collaboration according to claim 1 is characterized in that: The method comprises: when receiving a plurality of the user reasoning requests concurrently, dynamically adjusting the batch size of the reasoning task, and performing quantization processing on the data transmitted between the working nodes.

8. The large model reasoning method based on cross-domain edge device collaboration according to claim 7 is characterized in that: Dynamically adjust the batch size of the inference task: Calculating a ratio of a queue length to a batch size of the inference task; When the ratio is greater than 1, the batch size is increased; when the ratio is less than 0.5, the batch size is decreased.

9. A large model reasoning device based on cross-domain edge device collaboration, characterized in that: The device comprises: A processing unit, configured to dynamically sense available resource conditions of graphics processors in a plurality of working nodes based on a distributed hash table, and allocate large model blocks to the working nodes according to the available resource conditions; the available resource conditions represent a set of unallocated working nodes; A receiving unit, configured to receive a user reasoning request, wherein the reasoning request includes at least one or more reasoning tasks; A first allocation unit is used to allocate the reasoning task to one or more management nodes based on the large model blocks allocated to the working nodes; each area where the management node is located includes a set of working nodes; The second allocation unit is used for the management node to allocate the reasoning task to the working node set in the local area for reasoning; the working node set includes multiple working nodes.

10. A computer-readable storage medium storing one or more programs, wherein the one or more programs can be executed by one or more processors to implement the method according to any one of claims 1 to 8.

Images