Hardware reliability scheduling method and device for large model inference, equipment and medium

Abstract

This application relates to a hardware reliability scheduling method, apparatus, device, and medium for large-scale model inference. The method includes: non-intrusive instrumentation of a computing chip detection program before each inference task and between token calculations; collecting reliability data of each hardware component in the computing chip through the computing chip detection program; analyzing the reliability data of each hardware component in the computing chip to generate reliability analysis results for each hardware component; dynamically generating an optimal hardware scheduling strategy based on the reliability analysis results of each hardware component when executing the inference task; and executing the inference task through a large model based on the optimal hardware scheduling strategy. This application, through non-intrusive instrumentation, does not intrude on the user's inference task. It monitors the status of the computing chip based on hardware detection and scheduling capabilities, generates a more accurate hardware scheduling strategy for hardware scheduling to efficiently complete the inference task, and significantly improves the reliability and stability of large-scale model inference.

Description

Technical Field

[0001] This application relates to the field of large model application technology, and in particular to hardware reliability scheduling methods, devices, equipment and media for large model inference. Background Technology

[0002] As large-scale model technology moves from the technological exploration stage to industrial application, enterprises are facing three core challenges in promoting online inference and scenario-based deployment of large models: a complex model ecosystem, significant differences in hardware costs, and ever-changing scenario requirements. The main challenges lie in the significant reliability and cost differences between the Prefill and Decode stages, the varying reliability and cost of running the same model on different GPUs, the complexity of scenario parameters, and the differences in token generation and reliability requirements for different tokens. These issues make the stability of hardware operation a key bottleneck restricting the large-scale application of large models.

[0003] In current technologies, optimization solutions for large-scale model inference often focus on improving efficiency in a single dimension, resulting in a disconnect between reliability and business requirements. This fragmented optimization leads to GPU malfunctions during single-machine inference, making it difficult to reliably meet business reliability and service level requirements. Secondly, existing optimization solutions require intrusion into enterprise business code for instrumentation or extensive adaptation work at the business level, increasing deployment costs and maintenance complexity, and potentially disrupting the normal operation of existing business logic. This indirectly leads to hardware resource scheduling disorder, further reducing hardware reliability during large-scale model inference. Furthermore, current mainstream software-layer reliability solutions focus on fault handling at the software stack level, such as process restarts and node switching. They lack direct and effective detection methods for hardware-level failures of GPU core hardware components (such as SM Cores and GPU memory). Since SM Cores are the core units of GPU parallel computing and GPU memory is the key data storage carrier during inference, failures in either directly lead to inference task interruption and result distortion, representing a core manifestation of low hardware reliability.

[0004] It is evident that existing technologies suffer from low inference efficiency for large model inference due to low hardware reliability. Summary of the Invention

[0005] This application provides a hardware reliability scheduling method, apparatus, device, and medium for large model inference, in order to solve the technical problem of low inference efficiency caused by low hardware reliability in the prior art for large model inference.

[0006] According to one aspect of the embodiments of this application, this application provides a hardware reliability scheduling method for large-scale model inference. The method includes: performing non-intrusive instrumentation on a computing power chip detection program before each inference task and between token calculations; collecting reliability data of each hardware component in the computing power chip through the computing power chip detection program; analyzing the reliability data of each hardware component in the computing power chip to generate reliability analysis results for each hardware component; and dynamically generating an optimal hardware scheduling strategy based on the reliability analysis results of each hardware component when executing the inference task, and executing the inference task through a large model based on the optimal hardware scheduling strategy.

[0007] Optionally, before performing non-intrusive instrumentation on the computing chip detection program before each inference task and between token calculations, and before collecting reliability data of each hardware component in the computing chip through the computing chip detection program, the method further includes: viewing the interface definition file in the operating system based on a predetermined loading mechanism; rewriting the functions in the interface definition file based on the computing chip detection program to obtain a new interface definition file; compiling the new interface definition file into a shared object file, and setting a startup mechanism for the shared object file so that when the operating system starts, the shared object file compiled based on the computing chip detection program is loaded preferentially based on the startup mechanism.

[0008] Optionally, before performing non-intrusive instrumentation on the computing chip detection program before each inference task and between token calculations, and before collecting reliability data of each hardware component in the computing chip through the computing chip detection program, the method further includes: obtaining the operator computation frequency and running GPU modules of the current large model; constructing detection operators in the computing chip detection program and allocating GPU modules corresponding to the detection operators based on the operator computation frequency and running GPU modules of the current large model.

[0009] Optionally, the non-intrusive instrumentation of the computing chip detection program before each inference task and between token calculations, and the collection of reliability data of each hardware component in the computing chip through the computing chip detection program, includes: when the operating system starts, triggering the loading of the shared object file compiled based on the computing chip detection program based on the startup mechanism; parsing the specified library loaded preferentially based on the shared object file and preferentially parsing the functions in the specified library based on the dynamic linker; during the idle window period before running the inference task and between the token calculations, executing the computing chip detection program, collecting reliability data of the computing cores of the computing chip based on the detection operator and the specified library and functions loaded preferentially, and performing segmented scanning of the video memory based on a preset scanning strategy to collect reliability data of the video memory, wherein the reliability data of each computing core and the reliability data of the video memory are stored in shared video memory.

[0010] Optionally, the step of performing segmented scanning of the video memory based on a preset scanning strategy and collecting reliability data of the video memory includes: configuring a video memory scanning unit, writing segments of the video memory into the video memory under various data modes using the video memory scanning unit; comparing the data changes before and after writing, and determining the reliability data of the video memory based on the data changes before and after writing.

[0011] Optionally, the step of analyzing the reliability data of each hardware component in the computing chip to generate reliability analysis results for each hardware component includes: reading the reliability data of each computing core and the reliability data of the video memory from the shared video memory; calculating the reliability analysis results of the computing core based on the collected reliability data of each computing core in the computing chip using a computing core reliability evaluation function; and calculating the reliability analysis results of the video memory based on the collected reliability data of the video memory in the computing chip using a video memory reliability evaluation function.

[0012] Optionally, when executing the inference task, dynamically generating an optimal hardware scheduling strategy based on the reliability analysis results of each hardware component, and executing the inference task through a large model based on the optimal hardware scheduling strategy, includes: when executing the inference task, generating an optimal hardware scheduling strategy with the highest reliability of the computing chip as the objective, based on the reliability analysis results of the computing core and the video memory in the computing chip, wherein the optimal hardware scheduling strategy includes task allocation objects; when the large model runs the inference task, executing the inference task based on the task allocation objects allocated by the optimal hardware scheduling strategy.

[0013] According to another aspect of the embodiments of this application, this application provides a hardware reliability scheduling apparatus for large-scale model inference. The apparatus includes: a data acquisition module, used to perform non-intrusive instrumentation on a computing chip detection program before each inference task and between token calculations, and to acquire reliability data of each hardware component in the computing chip through the computing chip detection program; a data analysis module, used to analyze the reliability data of each hardware component in the computing chip and generate reliability analysis results for each hardware component; and a strategy generation module, used to dynamically generate an optimal hardware scheduling strategy based on the reliability analysis results of each hardware component when the inference task is executed, and to execute the inference task through a large model based on the optimal hardware scheduling strategy.

[0014] According to another aspect of the embodiments of this application, this application provides an electronic device, including: a processor, a memory, and a network interface. The memory stores machine-readable instructions executable by the processor. When the electronic device is running, the processor communicates with the memory through the network interface, and the processor executes the machine-readable instructions to perform the steps of the hardware reliability scheduling method for large model inference.

[0015] According to another aspect of the embodiments of this application, this application provides a storage medium having processor-executable non-volatile program code, the program code causing the processor to perform the steps of the hardware reliability scheduling method for large model inference.

[0016] Compared with related technologies, the technical solutions provided in this application have the following advantages:

[0017] The hardware reliability scheduling method for large-scale model inference provided in this application achieves complete isolation between the computing chip status monitoring and scheduling logic and user business code by providing a computing chip detection program independent of the user business chain. This fundamentally avoids business intrusion and makes the user unaware of the underlying monitoring and scheduling process. Furthermore, the computing chip detection program is lightweight, with its execution time compressed to the microsecond level. Compared to the inference task duration, its performance is nearly lossless, resulting in higher reliability and better suitability for the high-performance requirements of large-scale model inference. Program instrumentation is performed before each inference task and between token calculations. Based on the real-time detection of various hardware indicators in the computing chip by the computing chip detection program, reliability analysis is conducted. By deeply analyzing the underlying hardware indicators, a more accurate scheduling strategy can be generated, significantly improving the reliability and stability of large-scale model inference. Ultimately, this improved hardware reliability assists large models in efficiently completing inference tasks. Attached Figure Description

[0018] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application. To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, those skilled in the art can obtain other drawings based on these drawings without any creative effort.

[0019] Figure 1 This is a flowchart of an optional hardware reliability scheduling method for large model inference provided according to an embodiment of this application;

[0020] Figure 2 This is a schematic diagram of the runtime of an optional computing chip detection program provided according to an embodiment of this application;

[0021] Figure 3 This is a framework diagram of an optional scheduling system provided according to an embodiment of this application;

[0022] Figure 4 This is an optional large language model network structure diagram provided according to an embodiment of this application;

[0023] Figure 5 This is an architecture diagram of an optional large-scale computing power chip scheduling system provided according to an embodiment of this application;

[0024] Figure 6 This is a schematic diagram of an optional detection scan for the presence of zeros according to an embodiment of this application;

[0025] Figure 7 This is a schematic diagram of an optional detection scan for the presence of all-1 mode according to an embodiment of this application;

[0026] Figure 8 This is a schematic diagram of an optional detection scan of a visible checkerboard pattern according to an embodiment of this application;

[0027] Figure 9 This is a schematic diagram of an optional scanning method for detecting visible walking position patterns according to an embodiment of this application;

[0028] Figure 10 This is a schematic diagram of an optional scheduling process provided according to an embodiment of this application;

[0029] Figure 11 This is an optional module function interaction diagram provided according to an embodiment of this application;

[0030] Figure 12This is a flowchart illustrating an optional Slave and Master scheduling interaction according to an embodiment of this application.

[0031] Figure 13 This is a schematic diagram illustrating an optional scheduling result provided according to an embodiment of this application;

[0032] Figure 14 This is a schematic diagram of the structure of an optional hardware reliability scheduling device for large model inference according to an embodiment of this application;

[0033] Figure 15 This is a schematic diagram of an optional electronic device structure provided in an embodiment of this application. Detailed Implementation

[0034] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0035] To address the problems mentioned in the background art, according to one aspect of the embodiments of this application, an embodiment of a hardware reliability scheduling method for large model inference is provided.

[0036] It should be noted that the hardware reliability scheduling method for large model inference provided in this application is generally executed by a server and / or a terminal device, and correspondingly, the hardware reliability scheduling device for large model inference is generally set in the server / terminal device.

[0037] like Figure 1 As shown, Figure 1 A flowchart of a hardware reliability scheduling method for large model inference provided in this embodiment of the invention. Taking the execution of the hardware reliability scheduling method for large model inference by a server as an example, the hardware reliability scheduling method for large model inference includes the following steps:

[0038] Step S202: Non-intrusive instrumentation is performed on the computing power chip detection program before each inference task and between token calculations, and the reliability data of each hardware component in the computing power chip is collected through the computing power chip detection program.

[0039] In the context of large-model inference scenarios, computing power chips can refer to dedicated acceleration chips specifically designed for deep learning inference scenarios, possessing high parallel computing and high-bandwidth memory access capabilities. Their core function is to efficiently execute matrix operations, tensor calculations, and token generation processes for large models, replacing general-purpose CPUs to complete computationally intensive tasks. These include AI-specific acceleration chips and FPGA / ASIC inference chips for certain scenarios. In this embodiment, the computing power chip in the large-model inference service architecture can refer to a GPU. The GPU's hardware modules mainly include SM Cores (Streaming Multiprocessor Cores), CUDA Cores, video memory, control and scheduling modules, and communication modules. Among these, the Streaming Multiprocessor (SM) is the basic scheduling and execution cluster for GPU parallel computing and is the core functional unit of the GPU. It integrates sub-modules such as computing cores, caches, and execution units. A GPU contains multiple SMs, and the GPU's parallel capability is determined by the number of SMs and the performance of a single SM.

[0040] The computing chip detection program is independent of the large model inference program. Its insertion does not modify any code / binary files of the inference task. The detection program can be triggered by external mechanisms, such as dynamic linking, driver hijacking, and process injection. Therefore, instrumenting the computing chip detection program is non-intrusive and does not intrude on user business. The code and monitoring scheduling are completely isolated from user business, and users are completely unaware of the underlying monitoring and scheduling process.

[0041] Combination Figure 2 As shown, there is a certain idle time interval between adjacent inference tasks. Each inference task is processed in units of tokens during execution, and a single inference task will generate multiple tokens, with a small time interval between each token generation. Therefore, before executing an inference task, the computing power chip detection program is non-intrusively instrumented during the idle time interval before the inference task, and again during the idle time interval between token calculations.

[0042] Furthermore, upon receiving an inference request, before running the inference task, a millisecond-level (ms-level) computing chip detection program is executed first. Similarly, before generating a new token, a microsecond (µs-level) computing chip detection program is executed first. The difference between ms-level and microsecond-level programs arises because the amount of data involved in each stage of the detection differs. The computing chip detection program collects reliability data for each hardware component within the GPU. This reliability data allows analysis of the current state of the corresponding hardware, serving as an indicator for subsequent inference task allocation. Reliability data refers to data relevant to analyzing GPU reliability, including runtime status data, storage data, status and environment data, access and performance data, and error data.

[0043] Step S204: Analyze the reliability data of each hardware component in the computing chip to generate reliability analysis results for each hardware component.

[0044] This involves preprocessing the reliability data, including data cleaning and normalization. In this embodiment, analysis of the reliability data of each hardware component of the computing chip reveals whether the corresponding hardware reliability meets the user's business requirements. The reliability analysis results can be quantified using reliability scores, allowing for a more intuitive comparison of the reliability of each hardware component. Hardware with higher reliability scores can then be selected as the execution target for inference tasks. The reliability score (0-100 points) for each hardware module can be calculated using a preset model. Higher scores indicate stronger reliability. Different indicators can be assigned different weights, with higher weights having a greater impact on reliability. For example, core error indicators have the highest weight; if the ECC exceeds a preset value, 20 points are deducted for bit errors. Similarly, status indicators are scored based on deviation; 10 points are deducted when the SM temperature reaches 80℃-90℃, and 25 points are deducted above 90℃.

[0045] Furthermore, statistical analysis of reliability yields a score for each hardware module. For example, GPU SM0 Core scores 95 points, GPU0 memory scores 90 points; GPU SM1 Core scores 97 points, GPU1 memory scores 75 points. It's possible that the reliability analysis results can also generate risk warnings, highlighting core issues. For instance, a rising trend in single-bit ECC errors in GPU1 memory could trigger frequency throttling within 24 hours. Qualitative conclusions can also aid in strategy formulation by labeling hardware status, such as GPU SM0 Core being healthy, GPU1 memory exhibiting slight degradation, and GPU2 memory showing multiple bit errors, indicating high risk.

[0046] Step S206: When the inference task is executed, an optimal hardware scheduling strategy is dynamically generated based on the reliability analysis results of each hardware component, and the inference task is executed by a large model based on the optimal hardware scheduling strategy.

[0047] In this embodiment, the hardware modules are sorted in descending order based on the reliability analysis results. The modules ranked higher are prioritized for inference task execution. This means that an optimal hardware scheduling strategy is dynamically generated based on the reliability analysis results of each hardware module. Before executing the inference task, hardware modules with poor performance are replaced with more reliable ones. For example, if an inference task requires three SMs (Schedulers), before execution, the reliability scores of SM1, SM2, SM3, SM4, and SM5 are 85, 70, 90, 88, and 50 respectively. Before the inference task, inference is performed using SM1, SM2, and SM3. After reliability assessment and strategy scheduling, the task of SM2 is assigned to SM4. Finally, inference is performed using SM1, SM2, and SM4 to ensure higher overall hardware reliability. This results in higher stability and efficiency for the large model executing the inference task.

[0048] In this embodiment of the invention, by providing a computing chip detection program independent of the user's business chain, the computing chip status monitoring and scheduling logic is completely isolated from the user's business code. This fundamentally avoids business intrusion, making the user unaware of the underlying monitoring and scheduling process. Furthermore, the computing chip detection program is lightweight, with time compressed to the microsecond level. Compared to the inference task duration, its performance is nearly lossless, offering higher reliability and better suited to the high-performance requirements of large-scale model inference. Program instrumentation is performed before each inference task and between token calculations. Based on the real-time detection of various hardware indicators in the computing chip by the computing chip detection program, reliability analysis is conducted. By deeply analyzing the underlying hardware indicators, a more accurate scheduling strategy can be generated, significantly improving the reliability and stability of large-scale model inference. Ultimately, this improved hardware reliability assists large models in efficiently completing inference tasks.

[0049] In some optional embodiments, prior to step S202 described above, the method further includes:

[0050] S301, based on a predetermined loading mechanism, examines the interface definition files in the operating system;

[0051] S302, Based on the computing chip detection program, the functions in the interface definition file are rewritten to obtain a new interface definition file;

[0052] S303, the new interface definition file is compiled into a shared object file, and the startup mechanism of the shared object file is set so that when the operating system starts, the shared object file compiled based on the computing chip detection program is loaded first based on the startup mechanism.

[0053] Combination Figure 3 As shown, Figure 3This is an optional scheduling system framework diagram for the hardware reliability scheduling method for large model inference provided in this embodiment. Figure 3The architecture is structured as follows: At the very bottom: The CUDA driver core layer. CUDA Driver (Kernel Mode): This is the cornerstone of the entire architecture, running in the operating system kernel mode. It is responsible for directly interacting with the GPU hardware, handling core operations such as hardware resource scheduling and interrupts. CUDA Driver (User Mode): Provides user-mode driver interfaces, isolating the kernel-mode hardware details and allowing upper-layer code to safely call GPU functions. Driver API (CUDA): This is the programming interface for user-mode drivers, providing upper-layer runtimes and libraries with the basic ability to access the GPU. The middle layer: The CUDA runtime and supporting components. CUDARuntime (cudart.so): This is the core runtime library for CUDA programs, responsible for managing GPU context, memory allocation, stream scheduling, and other basic runtime services, simplifying the use of the GPU by developers. CUDA Libraries (cuBLAS.so): These are high-performance basic computing libraries provided by NVIDIA, encapsulating highly optimized matrix operations and other low-level computational logic for upper-layer frameworks to directly call to improve performance. SM Scheduler, Memory Manager, Metrics, and GlobalState are the core supporting components of the FlashInfer framework, belonging to the framework's underlying services. Specifically: SM Scheduler: Responsible for scheduling computational tasks on the GPU Streaming Multiprocessor (SM) to optimize computational resource utilization. Memory Manager: Manages the allocation and reclamation of GPU memory, reducing memory fragmentation and improving memory access efficiency. Metrics: Used to collect performance data, such as computation time and memory usage, providing a basis for performance tuning. GlobalState: Maintains the framework's global state information, ensuring consistency across multiple tasks or instances. At the top layer: The FlashInfer inference engine. FlashInfer is the top-level application of the entire architecture, a high-performance engine optimized for LLM inference. It calls basic libraries such as CUDA Runtime and cuBLAS and relies on its own supporting components to achieve efficient inference. Self Attention... is an optimization module in FlashInfer for core computations of large models. For example, the self-attention mechanism directly interacts with the framework's supporting components to achieve ultimate inference performance. Throughout the architecture, the underlying CUDA driver and runtime provide standardized GPU access capabilities, ensuring compatibility and stability. The supporting component in the middle layer is the FlashInfer performance engine, which enables the upper-layer inference logic to efficiently utilize GPU hardware through fine-grained scheduling, memory management, and state control. The top-level FlashInfer focuses on LLM inference scenarios, deeply binding complex computational logic with the underlying hardware capabilities to ultimately achieve high-throughput, low-latency inference performance.

[0054] In some examples, to achieve a non-intrusive scheduling system, the Linux preload mechanism can be used. By using a .so dynamic library, the entire scheduling system can be preloaded into the existing operating system at system startup using the command: export LD_PRELOAD= / mypath / bodesi_scheduler.so. This enables scheduling, memory management, monitoring metrics, and GlobalState storage to ensure the reliability of the computing cores and storage of all computing cards on the current machine.

[0055] In some examples, the principle and steps of rewriting CUDA's cuLaunchKernel via Preload mainly rely on the operating system's shared libraries (DLL format in Windows, SO format in Linux) to save video memory and disk space. During program execution, the system's "shared library registry" is checked to see if the required shared library is already in video memory. If it is, it is reused directly; if not, the shared library is read from disk and loaded into video memory, and simultaneously registered in the registry. The Preload mechanism, however, loads the relevant SO files into video memory before system startup, allowing for direct use without needing to load them from disk. This pre-loaded mechanism is called the Preload mechanism. Based on the Preload mechanism, the interface definition files in the operating system are first examined, and then functions are rewritten to obtain new interface definition files. Taking CUDA as an example, the various API definition files in the existing cuda.h file are examined, all functions in cuda.h are rewritten, and code for a computing chip detection program is inserted in the middle, as shown below:

[0056] cudaError_t cuLaunchKernel(

[0057] const void func, / / Kernel function entry address (function pointer)

[0058] dim3gridDim, / / Grid dimension (total number of blocks in the GPU kernel)

[0059] dim3blockDim, / / Thread block dimension (number of threads within each block)

[0060] void args, / / Array of kernel function arguments (an array of pointers, each element pointing to one argument)

[0061] size_tsharedMem, / / Shared memory size (bytes) for each thread block

[0062] cudaStream_tstream / / CUDA stream that executes the kernel function (default 0 indicates the default stream)

[0063] ){

[0064] / / Insert video memory allocation

[0065] Cudamalloc(size);

[0066] Cudahostmalloc(size);

[0067] / / Calling CUDA's cuLaunchKernel logic

[0068] The original cuLaunchKernel logic

[0069] / / Call copy

[0070] Cudacopyfromdevice();

[0071] }

[0072] Furthermore, in Linux systems, .so files refer to Shared Object files. Compiling a new interface definition file into a shared object file can be done by compiling the new interface definition file into a Linux .so file using the command: `gcc -fPIC -shared -o libhook.so -lm`. The startup mechanism is as follows: The `LD_PRELOAD` environment variable is set during operating system startup, allowing the API interfaces in the shared object file rewritten based on the computing chip detection program to be loaded preferentially. This includes both local and global preferential loading. Local preferential loading is specifically: `LD_PRELOAD / path / to / your / libhook.so . / your_program` #; This means that `libhook.so` is loaded preferentially only during the execution of `. / your_program`, affecting the current program and not other processes. The global preferential loading is: `export LD_PRELOAD= / path / to / your / libhook.so` #; This sets `LD_PRELOAD` as an environment variable for the current terminal session, ensuring that all programs launched from this terminal will preferentially load `libhook.so`. LD_PRELOAD is a very powerful environment variable in Linux systems. Its core function is to ensure that specified shared libraries are loaded preferentially when a program loads, and it can even override the system's default library functions. If a specified library contains a function with the same name as a system library, the program will preferentially call the version from the specified library, thereby replacing or enhancing the original functionality.

[0073] In this embodiment, functions are rewritten based on the Preload mechanism to obtain new interface definition files, and shared object files are compiled and the corresponding startup mechanism is configured. This ensures that the operating system prioritizes loading the interface set by the computing chip detection program at startup, without recompiling, restarting the system, or modifying the source code of the inference task. This achieves non-intrusive modification, completes the isolation between the monitoring and scheduling logic and the user business code, makes the user unaware of the underlying monitoring and scheduling process, and enables low-cost deployment.

[0074] In some optional embodiments, prior to step S202 described above, the method further includes:

[0075] S304 retrieves the operator computation frequency and the running GPU modules of the current large model;

[0076] S305, based on the computation frequency of the operators in the current large model and the running GPU modules, construct the detection operators in the computing power chip detection program and allocate the GPU modules corresponding to the detection operators.

[0077] In some examples, combined Figure 4 As shown, Figure 4 This embodiment provides an optional large language model network structure diagram. Input data is fed into the embedding layer, and after computation based on its intermediate layer, it is output to the output layer. Analysis reveals that the relevant operators include: Embedding operator: maps the input token ID to a high-dimensional dense vector (word embedding), requiring the loading of a pre-trained embedding matrix during inference. Positional encoding operator: adds spatiotemporal position information to the word embedding, with fixed encoding and learnable encoding options. Data type conversion operator: implements precision conversions such as FP32, FP16, FP8, and INT8, for example, converting weights from FP32 to FP16 / INT8 during inference to reduce memory usage and computational load. Reshape / Transpose operator: adjusts tensor dimensions, such as converting [batch_size, seq_len, d_model] to [batch_size × seq_len, d_model] and swapping axis order to adapt to the input format of subsequent calculations. Slice / Split / Concat operators: Slice extracts a portion of a tensor, Split splits a tensor along its dimensions, and Concat joins multiple tensors together. Cast operator: Implements type casting of tensor data types, ensuring data type compatibility between different operators; for example, Softmax requires floating-point input.

[0078] In this embodiment, different operators call different types of GPU modules, such as MatMul calling TensorCore and Attention calling SM+ memory controller. Furthermore, the operator computation frequencies vary significantly. To achieve more accurate, lightweight, and efficient GPU hardware reliability detection, the detection operators required by the aforementioned computing power chip detection program can be designed and corresponding GPU modules allocated by statistically analyzing the computation frequencies of the operators used in the current large model and obtaining the GPU modules used in the current large model. The specific details are shown in the table below:

[0079] Table 1. Detection Operator Types and Corresponding GPU Modules

[0080]

[0081] The design and result judgment for each detection operator are shown in the table below:

[0082] ReLU operator (Rectified Linear Unit): Calculates max(x,0) element-wise. Outputs 0 if the input is less than 0, otherwise outputs the original value. See Table 2 below:

[0083] Table 2 ReLU Operators

[0084]

[0085] The Sigmoid operator calculates 1 / (1+exp(-x)) element-wise, with an output range of (0,1). See Table 3 below:

[0086] Table 3 Sigmoid Operator

[0087]

[0088] Multiplication (Mul) operator: Calculates the product of two tensors element-wise (supports broadcasting). See Table 4 below:

[0089] Table 4 Multiplication (Mul) Operators

[0090]

[0091] Division (Div) operator: Calculates the quotient of two tensors element-wise (supports broadcasting mechanism), but division by zero must be avoided. See Table 5 below:

[0092] Table 5 Division (Div) Operators

[0093]

[0094] Matrix multiplication (GEMM) operator: Calculates the product of two matrices (dimension-adapted: A[M,K]×B[K,N]=C[M,N]). See Table 6 below:

[0095] Table 6 Matrix Multiplication (GEMM) Operators

[0096]

[0097] Quantize / Dequantize operators: Quantize converts an FP32 tensor to INT8 / INT16; Dequantize restores a low-precision tensor to FP32. See Table 7 below:

[0098] Table 7 Quantize / Dequantize Operators

[0099]

[0100] TopK operator: Filters the top K maximum or minimum values ​​from a tensor according to its dimensions. See Table 8 below:

[0101] Table 8 TopK Operators

[0102]

[0103] In this embodiment, detection operators in the computing power chip detection program are constructed by combining the operator computation frequency of the current large model with the GPU module usage, and matched with the modules in the GPU to avoid indiscriminate detection. This focuses on the actual hardware that the operators depend on, and allows for hierarchical design of detection granularity based on the computation frequency of the detection operators, balancing detection effectiveness and inference performance. This ensures accurate acquisition of reliability data from each GPU module without increasing inference latency of large models due to excessive computing power consumption by detection, thus better meeting the system's SLO requirements.

[0104] In some optional embodiments, step S202 above includes:

[0105] S2021, When the operating system starts, the shared object file compiled based on the computing chip detection program is loaded based on the startup mechanism;

[0106] S2022, Based on the dynamic linker, the specified library is loaded first in the shared object file and the functions in the specified library are parsed first.

[0107] S2023, during the idle window period before running the inference task and between the token calculation, the computing chip detection program is executed. Based on the detection operator and the specified library and function loaded in priority, the reliability data of the computing core of the computing chip is collected, and the video memory is scanned in segments based on a preset scanning strategy to collect the reliability data of the video memory. The reliability data of each computing core and the reliability data of the video memory are stored in shared video memory.

[0108] Combination Figure 5 As shown, Figure 5This paper presents an architecture diagram of an optional large-model computing chip scheduling system based on the hardware reliability scheduling method for large-model inference provided in this application. The overall logic is a closed-loop process of user request - dynamic instrumentation - scheduling control - kernel monitoring - hardware linkage, which can provide real-time reliability detection and resource scheduling guarantee for the operation of large-model operators. The user request path is: user request, VLLM (large-model inference framework), FlashAttention (efficient attention mechanism), and computing chip user-space driver, serving as the entry point for business traffic, processing large-model inference requests, and finally distributing the computing tasks to the computing chip. Dynamic instrumentation detection: In the token calculation stage based on FlashAttention, the system injects the computing chip detection program into the calculation process through dynamic instrumentation. The detection program generates an eBPF program using tools such as Clang / Bpftool and dynamically mounts it into the execution link of the user-space driver, achieving non-intrusive monitoring of the large-model operator runtime. The scheduling layer includes: Scheduler, Memory Manager, Global State, and Metrics. The Scheduler and Memory Manager are responsible for the global scheduling of computing tasks and memory resources of the computing chip, allocating resources based on real-time status. Global State maintains the resource and task status of the entire cluster and serves as the basis for scheduling decisions. Metrics collects hardware performance metrics from the kernel in real time, providing data support for scheduling and reliability assessment. The kernel layer includes the detection program sharing memory and the computing chip driver. The computing chip driver directly interacts with the hardware, executes computing tasks, and reports GPU hardware performance data to the computing chip detection program—essentially collecting and reporting reliability data. The collected reliability data from the detection operators and memory is written to the shared memory for the scheduling layer's Metrics module to read, enabling data exchange between user space and kernel space.

[0109] The computing cores include SM, CUDA Core, Tensor Core, Warp scheduler, registers, etc. In this embodiment, the reliability data of the aforementioned computing cores may include, but is not limited to, the hardware status, execution behavior, faults / errors, and environmental parameters of the computing core SMcore, such as instruction execution count, number of computation errors, core temperature, voltage, frequency, SM utilization, ECC check errors, and number of frequency reductions. The reliability data of the aforementioned video memory includes, but is not limited to, video memory bandwidth utilization, average / maximum access latency, memory access conflict frequency, cache miss rate, read / write traffic ratio, video memory chip / controller temperature, video memory power supply voltage / current fluctuation values, video memory channel online status, ECC function enabled status, remaining available video memory, and video memory fragmentation degree, etc.

[0110] In this embodiment, based on the aforementioned scheduling framework, after the code of the computing chip detection program is inserted, during the operating system startup phase, the user executes a command, triggering the loading of the compiled .so file based on the startup mechanism, for example, LDPRELOAD= / path / libmycuda.so vllm. During the linking phase, the dynamic linker resolves LD PRELOAD to preferentially load specified libraries, for example, loading interfaces such as cuLaunchKernel. Specifically, the dynamic linker resolves the specified libraries that the program depends on during program runtime, loads the library files into the process address space, and completes the mapping and binding of function symbols, allowing the program to call functions in the specified libraries. Further, the dynamic linker performs symbol resolution, overriding function symbols with the same name by using function symbols from the Preload library, for example, overriding the original cuLaunchKernel interface. Finally, the program runs normally, and the logic of the Preload library is injected and executed, for example, calling the overridden cuLaunchKernel interface.

[0111] Furthermore, by executing a detection program before the system performs inference tasks and during the idle window between network computing tokens, reliability data of the computing chip's core can be collected based on the designed detection operators and the specified libraries and functions that are loaded preferentially. Simultaneously, the computing chip's video memory can be scanned using a preset scanning strategy to collect video memory reliability data. The collected reliability data can be categorized and stored in shared video memory for the scheduling layer's Metrics module to read.

[0112] In this embodiment, during the operating system startup phase, the shared object file compiled by the detection program is loaded through the startup mechanism. Combined with the dynamic linker's priority parsing of specified libraries and functions, this ensures that the detection logic is deployed before the inference task is executed and does not tamper with the core code of the inference program. The detection program is executed before the inference task runs and during the idle window of token calculation. It not only uses the detection operator to accurately collect and calculate the core reliability data, but also obtains the video memory reliability data through segmented scanning. All data is stored in shared video memory. This avoids the problem of the detection logic preempting the core computing power of inference and increasing the token generation delay. It also realizes the real-time collection and unified storage of hardware reliability data, providing a highly timely and targeted hardware status basis for the generation of dynamic scheduling strategies, while ensuring the SLO of the large model inference service.

[0113] In some optional embodiments, step S2023 above, which involves segmenting the video memory based on a preset scanning strategy and collecting reliability data of the video memory, includes:

[0114] Configure the video memory scanning unit, and write the video memory into segments under various data modes according to the video memory scanning unit;

[0115] By comparing the data changes before and after the write operation, the reliability data of the video memory is determined based on the data changes before and after the write operation.

[0116] Specifically, video memory itself is a contiguous region. To enable rapid detection, the video memory scan unit can be preset, which can be configured at system startup. Both the video memory scan unit and the number of writes can be adjusted; no single limitation is set here.

[0117] Combination Figures 6 to 9 As shown, in some examples, the video memory is scanned in segments of 4MB, such as... Figure 6 The diagram shows the all-zero mode: 4KB of video memory is written to a data pattern consisting entirely of zeros 10 times, and then the data is compared. If some bits are not zero, it indicates a problem. This can be used to verify whether the video memory can stably maintain a low level. The bit data situation in the all-zero mode is also one of the indicators of video memory reliability.

[0118] like Figure 7 As shown, in the all-1 mode, 4KB of video memory is written with all 1s 10 times. If some bits are not 1, it indicates a problem. This method can be used to test the video memory's ability to maintain a high level, which is also an evaluation indicator of video memory reliability.

[0119] like Figure 8As shown, this is the checkerboard pattern: each detection involves periodically writing 0s and 1s every 4MB (1ms), for a total of 5 writes. After each crossover, the data is verified to be the same as the written data to detect adjacent bit interference. If they are different, adjacent bit interference is considered to exist; otherwise, it does not. This metric can also be used as one of the data points for memory reliability.

[0120] like Figure 9 As shown, this is the bit-shifting mode: Each shift detects only one bit as 1, with the rest as 0, shifting bit by bit. The presence or absence of a change in position after each shift determines whether there is an error in the video memory, allowing for precise location of individual bit line faults. If there is a change after the shift, it indicates a video memory error; otherwise, the video memory is error-free, and this is also used as one of the video memory reliability indicators.

[0121] In this embodiment, by comparing the changes before and after data writing under different data modes, it is possible to determine whether there are problems with the video memory in various aspects based on the changes before and after writing. This indicator can be used as a basis for evaluating the reliability of the video memory. Furthermore, by performing tests under multiple data modes, the video memory can be tested from multiple dimensions, thereby improving the accuracy of the video memory reliability judgment.

[0122] In some optional embodiments, step S204 above includes:

[0123] S2041, Read the reliability data of each computing core and the reliability data of the video memory from the shared video memory;

[0124] S2042, Based on the computing core reliability evaluation function, the reliability data of each computing core in the collected computing chip is used to calculate and obtain the reliability analysis result of the computing core.

[0125] S2043, based on the memory reliability evaluation function, calculates the reliability analysis result of the memory by means of the collected memory reliability data in the computing chip.

[0126] In some examples, the Metrics component parses relevant reliability information from the shared memory of the detection program, summarizes, calculates, and trains the data. Then, it loads the reliability information of the computing cores, video memory, and other components of the computing chip into GlobalState. In other words, it loads the reliability data of each computing core and video memory read from the shared video memory into GlobalState for storage. When a user makes an inference request, the Scheduler determines which computing cores and video memory to use to handle the request to achieve optimal reliability based on the reliability data of the computing cores and video memory in GlobalState. That is, it generates the optimal hardware scheduling strategy based on the reliability data of the computing cores and video memory in GlobalState.

[0127] For the detection operator, the calculation result is divided into two types: detection failure, the reliability of the corresponding calculation core is 0; if the test result is not 0, the reliability score of the calculation core is evaluated by the calculation core reliability evaluation function, as shown in the following formula (1):

[0128] (1)

[0129] in, S Represents the reliability score, ranging from [0, 100]. F Indicates the frequency of operator operations. F max Indicates the maximum frequency at which the GPU can operate safely. R Indicates the error count. R max This represents the maximum permissible error threshold for the computational core. N This indicates the total number of times the operator is run. N max Indicates the maximum number of permissible operations (times) within the design life. α, β, γ These are the weighting coefficients for each corresponding indicator, and their sum is 1. k This indicates the risk coefficient.

[0130] The physical meaning of each parameter is as follows: computing power chip S Used to comprehensively evaluate the long-term reliability of the detection operator, 100 points indicates that there is theoretically no risk of failure, and 0 points indicates that failure is inevitable. F Replace with the current operating frequency of the GPU; F max This is the manufacturer's stated maximum acceleration frequency; exceeding this value significantly increases the risk. N This indicates the cumulative number of executions per 1ms. N max Estimation is performed based on the maximum frequency and computational cost. α, β, γ This indicates adjusting the influence weights of various factors, including computationally intensive operators and storage-intensive operators. k This represents the overall scaling factor, which is adjusted according to the GPU process, such as 1.5 for 7nm process and 2.0 for 14nm process. Based on the above equation (1), the reliability analysis results of each computing core can be calculated by substituting the parameters, which is the reliability score.

[0131] Similarly, based on the following formula (2) memory reliability evaluation function, the reliability analysis results of the memory can be calculated.

[0132] (2)

[0133] in, S memRepresents the memory reliability score, ranging from [0, 100]. f Indicates the current memory frequency. f max Indicates the maximum rated frequency of the memory. T Indicates the current operating temperature of the memory. T ref Indicates the reference operating temperature of the memory. R total R represents the total number of 0 and 1 inversions. max_total Indicates the maximum number of reversals (bits) allowed within the design life. α, β, γ This represents the weighting coefficient of the corresponding indicator. k This indicates the risk coefficient.

[0134] The physical meaning of each parameter is as follows: S mem This represents a comprehensive assessment of the long-term reliability of the memory; 100 indicates theoretically no failures, and 0 indicates inevitable failure. f Replace with the current memory operating frequency of the graphics card; f max It is determined by the memory specification; for example, 16Gbps for GDDR6 corresponds to 2GHz. R total This indicates the total number of bits ranging from 0 to 1 and 1 to 0 during memory read / write operations; R max_total Calculations based on memory wear and tear, determined according to the memory manufacturer's specifications. α, β, and γ are adapted to memory type: DRAM is emphasized (critical for refresh); NAND flash memory is emphasized (wear is triggered by inversion). k Process compatibility: 1.2 for advanced processes, 2.0 for older processes. T corresponds to the real-time temperature of the memory chip or module; T ref The standard temperature used for manufacturer testing is usually 25°C or 85°C. After substituting the parameters into equation (2) above, the reliability analysis results of the video memory can be calculated.

[0135] In some optional embodiments, to improve operational efficiency during the detection process, the execution frequency of detection test cases can be reduced through dynamic configuration. Currently, the execution frequency of detection test cases can be adjusted according to the number of requests and the interval of the number of tokens generated. The recording and identification of the frequency are as follows: Request identification: Request identification is the calculation of a request between the embedding operator and the output operator. In this embodiment, the number of requests is represented by recording the execution count of the embedding operator + the output operator, and the detection frequency is based on the interval of the requests. Token identification: When the model completes "probability output → token selection", for example, when the "park" token is selected, it marks the end of one token and the beginning of the next token.

[0136] In this embodiment, reliability data is read directly from shared video memory. By leveraging the high bandwidth of shared video memory, the latency loss caused by data acquisition across PCIe links is avoided, ensuring the real-time performance and integrity of hardware status data. Reliability quantification calculation is performed based on the corresponding evaluation function to obtain the reliability analysis results of each computing core and the video memory, thereby achieving accurate quantification and horizontal comparability of the health status of computing cores and video memory.

[0137] In some optional embodiments, step S206 above includes:

[0138] S2061, when the inference task is executed, the optimal hardware scheduling strategy with the highest reliability of the computing chip as the objective is generated based on the reliability analysis results of the computing core and the video memory in the computing chip, wherein the optimal hardware scheduling strategy includes task objects to be assigned.

[0139] S2062, When the large model runs an inference task, the task to be allocated based on the optimal hardware scheduling strategy executes the inference task.

[0140] In some embodiments, the specific storage structure in GlobalState is as follows:

[0141] UUID: xxx-xxx-xxx

[0142] CUDA score: [<coreid, bit> ,<coreid, bit> , ……,<coreid, bit> ]

[0143] tensor score: [<coreid, bit> ,<coreid, bit> , ……,<coreid, bit> ]

[0144] mem score: [<memid, bit> ,<memid, bit> , ……,<memid, bit> ]

[0145] Each field represents the following: Uuid: The GPU ID, unique across the entire machine; CUDA core: An array of IDs and reliability scores for each CUDA core; Tensor score: An array of IDs and reliability scores for each tensor core; Mem score: An array of IDs and reliability scores for each memory segment. After summarizing the GlobalState, the Launchkernel function is called to run the specific functions when the large model is executed.

[0146] Combination Figures 10-11 As shown, in some examples, to match the optimal hardware scheduling strategy, the Scheduler first filters out suitable cores and suitable memory regions, and the scheduling strategy adopts a master-slave structure. Figure 10 In the diagram, the one marked with "△" is the Scheduler master (or simply master, the central hub for unified scheduling and management), and the one marked with "○" is the Scheduler slave (or slave, the execution node, the actual executor of tasks, and the one that reports resources). Figure 11 In this module, the functions of the two modules are as follows: Scheduler Master: A dedicated SM Core deployed on the GPU, reserving 1-2 SMs as scheduling cores, responsible for internal task scheduling on the GPU; Scheduler Slave: Each computing SM Core has built-in Slave logic, implemented through GPU kernel functions, and communicates directly with the Master through GPU shared memory.

[0147] Combination Figure 12 As shown, to more clearly illustrate the working process of the two modules, taking the Nvidia GPU as an example, the scheduling on the GPU is first initialized: the initialization parameters (number of SMs, reliability baseline, scheduling policy) are written to the GPU's global constant memory through cudaMemcpyToSymbol; GPU internal initialization: the scheduling-dedicated SM starts the Master kernel, scans all compute SMs and builds an SM list; each compute SM starts the Slave kernel and initializes the local monitoring logic; the Master and Slave establish a communication channel through GPU shared memory (pre-allocated fixed address space).

[0148] Furthermore, combined Figure 12 As shown, the Slave and Master scheduler interact:

[0149] During the task submission phase, the CPU (task submitter) writes batch task data and task metadata to the GPU shared memory, and then sends a single batch task ready signal to the GPU scheduler (Master), triggering the scheduling process. During the scheduling and allocation phase, the GPU scheduler (Master) reads task metadata from the shared memory and, based on the SM reliability database, selects the GPU computing SMs (Slaves) with higher reliability as the target execution units. After determining the target, the scheduler (Master) writes the "SM-ID → Task ID" mapping relationship into the task allocation table in the shared memory. Each GPU computing SM reads this allocation table in a round-robin fashion to obtain the task assigned to it. During the execution and monitoring phase, while executing tasks, the GPU computing SM starts local status monitoring, writing task progress and its own hardware status (such as load and error count) to the shared memory in real time. The GPU scheduler (Master) continuously monitors the task status in the shared memory, tracking the overall execution progress. During the exception handling and task migration phase, when a GPU computing SM encounters an exception, it proactively writes an exception marker to the shared memory. When the GPU scheduling SM detects an anomaly, it triggers local decision-making task migration logic, updates the task allocation table in shared memory, and reassigns the incomplete tasks of the anomaly SM to other healthy Slave SMs. After other Slave SMs poll the updated allocation table, they take over and execute the migration tasks, that is, select the more reliable Slave SM to perform the inference task. In the result return phase, after all tasks are completed, the GPU computing SM writes the task results to shared memory and sends a single-result ready signal to the CPU. Upon receiving the signal, the CPU reads the final task result from shared memory through the GPU scheduling SM, completing the entire process.

[0150] like Figure 13 As shown, taking the GPU's SM Core computing core as an example, the operating system running based on the hardware reliability scheduling method for large model inference provided in this embodiment can ensure that when the SM Core has poor reliability, it can use a more reliable SM Core for computation after the next switch, so as to achieve high reliability of inference. That is, it can find the optimal hardware scheduling strategy to execute the inference task with the locked task to be assigned. If an anomaly is detected in one of the currently running SMs, and the reliability is poor ( Figure 13 (SM3 contains black blocks). To address this, the inference task that was originally assigned to SM3 containing black blocks is moved to SM5, which does not contain any anomalies, through scheduling.

[0151] According to another aspect of the embodiments of this application, such as Figure 14As shown, corresponding to the hardware reliability scheduling method for large model inference in the above embodiments, this embodiment provides a hardware reliability scheduling device for large model inference, the device comprising:

[0152] The data acquisition module 501 is used to perform non-intrusive instrumentation on the computing chip detection program before each inference task and between token calculations, and to collect reliability data of each hardware component in the computing chip through the computing chip detection program.

[0153] The data analysis module 503 is used to analyze the reliability data of each hardware component in the computing chip and generate reliability analysis results for each hardware component.

[0154] The strategy generation module 505 is used to dynamically generate an optimal hardware scheduling strategy based on the reliability analysis results of each hardware when the inference task is executed, and to execute the inference task based on the optimal hardware scheduling strategy through a large model.

[0155] It should be noted that in this embodiment, the data acquisition module 501 can be used to execute step S202 in this application embodiment, the data analysis module 503 in this embodiment can be used to execute step S204 in this application embodiment, and the strategy generation module 505 in this embodiment can be used to execute step S206 in this application embodiment.

[0156] According to another aspect of the embodiments of this application, a computer program product or computer program is also provided, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the steps of the hardware reliability scheduling method for large model inference in any of the above embodiments.

[0157] According to another aspect of the embodiments of this application, this application also provides an electronic device, such as... Figure 15 As shown, the device includes a memory 601, a processor 603, and a network interface 605. The memory 601 stores a computer program that can run on the processor 603. The memory 601 and the processor 603 communicate through the network interface 605 and a communication bus 607. When the electronic device is running, the processor 603 and the memory 601 communicate through the network interface 605. When the processor 603 executes the computer program, it implements the steps of the hardware reliability scheduling method for large model inference described above.

[0158] The memory and processor in the aforementioned electronic device communicate with each other via a communication bus and a communication interface. The communication bus can be a peripheral component interconnect standard (PCI) bus or an extended industry standard structure (EISA) bus, etc. This communication bus can be divided into an address bus, a data bus, a control bus, etc. The memory can include random access memory (RAM) or non-volatile memory, such as at least one disk storage device. Optionally, the memory can also be at least one storage device located remotely from the aforementioned processor. The aforementioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0159] It should be noted that, in this document, relational terms such as first, second, etc., are used only to distinguish one entity or operation from another entity or operation. The terms include, encompass, or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0160] The above description is merely a specific embodiment of this application, enabling those skilled in the art to understand or implement this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features claimed herein.

Claims

1. A hardware reliability scheduling method for large model inference, characterized in that, The method includes: Non-intrusive instrumentation is performed on the computing chip detection program before each inference task and between token calculations, and the reliability data of each hardware component in the computing chip is collected through the computing chip detection program. Based on the reliability data of each hardware component in the computing chip, the reliability analysis results of each hardware component are generated. When the inference task is executed, an optimal hardware scheduling strategy is dynamically generated based on the reliability analysis results of each hardware component, and the inference task is executed by a large model based on the optimal hardware scheduling strategy. Before performing non-intrusive instrumentation on the computing chip detection program before each inference task and between token calculations, and before collecting reliability data of each hardware component in the computing chip through the computing chip detection program, the method further includes: viewing the interface definition file in the operating system based on a predetermined loading mechanism; rewriting the functions in the interface definition file based on the computing chip detection program to obtain a new interface definition file; compiling the new interface definition file into a shared object file, and setting a startup mechanism for the shared object file so that when the operating system starts, the shared object file compiled based on the computing chip detection program is loaded preferentially based on the startup mechanism.

2. The hardware reliability scheduling method for large model inference of claim 1, wherein, Before performing non-intrusive instrumentation on the computing chip detection program before each inference task and between token calculations, and before collecting reliability data of each hardware component in the computing chip through the computing chip detection program, the method further includes: Obtain the operator computation frequency and the running GPU modules of the current large model; Based on the computation frequency of the operators in the current large model and the running GPU modules, the detection operators in the computing power chip detection program are constructed and the corresponding GPU modules are allocated.

3. The hardware reliability scheduling method for large model inference of claim 2, wherein, The non-intrusive instrumentation of the computing chip detection program before each inference task and between token calculations, and the collection of reliability data of each hardware component in the computing chip through the computing chip detection program, including: When the operating system starts, the shared object file compiled based on the computing chip detection program is loaded based on the startup mechanism. Based on the dynamic linker, the shared object file is parsed and the specified library is loaded first, and the functions in the specified library are parsed first. During the idle window period before running the inference task and between the token calculation, the computing chip detection program is executed. Based on the detection operator and the specified library and function loaded in priority, the reliability data of the computing core of the computing chip is collected, and the video memory is scanned in segments based on a preset scanning strategy to collect the reliability data of the video memory. The reliability data of each computing core and the reliability data of the video memory are stored in shared video memory.

4. The hardware reliability scheduling method for large model inference according to claim 3, characterized in that, The method of performing segmented scanning of the video memory based on a preset scanning strategy and collecting reliability data of the video memory includes: Configure the video memory scanning unit, and write the video memory into segments under various data modes according to the video memory scanning unit; By comparing the data changes before and after the write operation, the reliability data of the video memory is determined based on the data changes before and after the write operation.

5. The hardware reliability scheduling method for large model inference according to claim 3, characterized in that, The reliability data of each hardware component in the computing chip is analyzed to generate reliability analysis results for each hardware component, including: Read the reliability data of each computing core and the reliability data of the video memory from the shared video memory; Based on the core reliability evaluation function, the reliability data of each core in the collected computing chip are used to calculate the reliability analysis results of the core. Based on the memory reliability evaluation function, the reliability analysis results of the memory are obtained by calculating the reliability data of the memory in the collected computing chip.

6. The hardware reliability scheduling method for large model inference according to claim 5, characterized in that, When executing the inference task, an optimal hardware scheduling strategy is dynamically generated based on the reliability analysis results of each hardware component. The inference task is then executed using a large model based on this optimal hardware scheduling strategy, including: When the inference task is executed, the optimal hardware scheduling strategy is generated based on the reliability analysis results of the computing core in the computing chip and the reliability analysis results of the video memory, with the highest reliability of the computing chip as the objective. The optimal hardware scheduling strategy includes the task to be assigned. When a large model runs an inference task, the task to be assigned is executed by the task to be assigned based on the optimal hardware scheduling strategy.

7. A hardware reliability scheduling apparatus for large model inference, used to implement the hardware reliability scheduling method for large model inference as described in any one of claims 1 to 6, characterized in that, The device includes: The data acquisition module is used to perform non-intrusive instrumentation on the computing chip detection program before each inference task and between token calculations, and to collect reliability data of each hardware component in the computing chip through the computing chip detection program. The data analysis module is used to analyze the reliability data of each hardware component in the computing chip and generate reliability analysis results for each hardware component. The strategy generation module is used to dynamically generate an optimal hardware scheduling strategy based on the reliability analysis results of each hardware when the inference task is executed, and to execute the inference task based on the optimal hardware scheduling strategy through a large model.

8. An electronic device, comprising: A processor, a memory, and a network interface, wherein the memory stores machine-readable instructions executable by the processor, characterized in that: when the electronic device is running, the processor communicates with the memory via the network interface, and the processor executes the machine-readable instructions to perform the steps of the hardware reliability scheduling method for large model inference as described in any one of claims 1 to 6.

9. A storage medium having processor-executable non-volatile program code, characterized in that, The program code causes the processor to execute the steps of the hardware reliability scheduling method for large model inference as described in any one of claims 1 to 6.

Images