A large language model operator precision verification method and device for heterogeneous hardware

Abstract

The embodiment of the application discloses a large language model operator precision verification method and device for heterogeneous hardware, comprising: after deploying a large language model on heterogeneous hardware, obtaining first intermediate hidden states, first key cache states and first value cache states output by each network layer when the large language model calls each operator kernel to execute autoregressive inference; obtaining second intermediate hidden states, second key cache states and second value cache states output by each network layer when the large language model calls each operator kernel to execute pre-padding inference; calculating difference values between each state output by the same network layer under the same operator kernel; and determining whether an operator set corresponding to each network layer exists precision anomaly when the large language model calls each operator kernel to execute operation based on each difference value and a preset difference threshold, so that the operator precision of the large language model deployed on the heterogeneous hardware can be accurately verified.

Description

Technical Field

[0001] This invention relates to the field of operator precision verification, and in particular to a method and apparatus for verifying the precision of large language model operators for heterogeneous hardware. Background Technology

[0002] With the rapid development of large language models, model inference has been widely deployed on various heterogeneous hardware. However, due to the significant differences between heterogeneous hardware architecture and homogeneous hardware architecture such as general-purpose graphics processing units (GPUs) and central processing units (CPUs), standard operators cannot usually be run directly. It is necessary to redesign and implement an equivalent operator library for the characteristics of heterogeneous hardware.

[0003] Currently, to verify the operator accuracy of large language models deployed on heterogeneous hardware, the operator output obtained from deploying the large language model on homogeneous hardware is generally used as the baseline truth value and compared with the operator output of heterogeneous hardware. The operator accuracy is then determined based on the comparison results. However, due to architectural differences such as different accumulator precision, heterogeneous and homogeneous hardware are prone to deviations in numerical distribution, making direct comparison of the operator outputs of the two types of hardware difficult. Secondly, in the whole-network testing of large language models, if the final output accuracy is low, it is difficult to distinguish whether it is due to accumulated error, sampling strategy issues, or implementation errors of specific operators. Finally, in long sequence inference, due to differences in batch processing methods and key-value cache management mechanisms, the intermediate layer states of heterogeneous hardware are difficult to strictly align with the intermediate layer states of homogeneous hardware in terms of time steps and memory layout, making it impossible to perform network layer-by-network comparisons, thus making it difficult to accurately verify the operator accuracy of large language models deployed on heterogeneous hardware. Summary of the Invention

[0004] This invention provides a method and apparatus for verifying the operator precision of a large language model on heterogeneous hardware. It solves the problem that existing technologies make it difficult to accurately verify the operator precision of a large language model deployed on heterogeneous hardware due to differences in architecture, operator output precision deviation, batch processing methods, or key-value cache management mechanisms. This invention can accurately verify the operator precision of a large language model deployed on heterogeneous hardware.

[0005] In a first aspect, embodiments of the present invention provide a method for verifying the precision of operators in a large language model for heterogeneous hardware, comprising: after deploying a large language model on heterogeneous hardware, obtaining a first intermediate hidden state, a first key cache state, and a first value cache state output by each network layer when the large language model calls each operator kernel to perform autoregressive inference; different operator kernels correspond to different hardware instructions; obtaining a second intermediate hidden state, a second key cache state, and a second value cache state output by each network layer when the large language model calls each operator kernel to perform pre-filled inference; calculating the difference between the first intermediate hidden state and the second intermediate hidden state, the difference between the first key cache state and the second key cache state, and the difference between the first value cache state and the second value cache state output by the same network layer under the same operator kernel; and determining, based on each difference value and a preset difference threshold, whether there is a precision anomaly in the operator set corresponding to each network layer when the large language model calls each operator kernel to perform operations.

[0006] Secondly, embodiments of the present invention also provide a large language model operator precision verification device for heterogeneous hardware, comprising: a first state acquisition module, used to acquire, after deploying the large language model on heterogeneous hardware, a first intermediate hidden state, a first key cache state, and a first value cache state output by each network layer when the large language model calls each operator kernel to perform autoregressive inference; different operator kernels correspond to different hardware instructions; a second state acquisition module, used to acquire, when the large language model calls each operator kernel to perform pre-filled inference, a second intermediate hidden state, a second key cache state, and a second value cache state output by each network layer; a difference value determination module, used to calculate the difference value between the first intermediate hidden state and the second intermediate hidden state, the difference value between the first key cache state and the second key cache state, and the difference value between the first value cache state and the second value cache state output by the same network layer under the same operator kernel; and an anomaly identification module, used to determine, based on each difference value and a preset difference threshold, whether there is a precision anomaly in the operator set corresponding to each network layer when the large language model calls each operator kernel to perform operations.

[0007] Thirdly, embodiments of the present invention also provide an electronic device, the electronic device comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores a computer program executable by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to execute the large language model operator precision verification method for heterogeneous hardware provided in any embodiment of the present invention.

[0008] Fourthly, embodiments of the present invention also provide a computer-readable storage medium storing computer instructions for causing a processor to execute and implement the large language model operator precision verification method for heterogeneous hardware provided in any embodiment of the present invention.

[0009] The technical solution of this invention, by deploying a large language model on heterogeneous hardware, obtains the first intermediate hidden state, the first key cache state, and the first value cache state output by each network layer when the large language model calls each operator kernel to perform autoregressive inference; obtains the second intermediate hidden state, the second key cache state, and the second value cache state output by each network layer when the large language model calls each operator kernel to perform pre-filling inference; calculates the difference between the first and second intermediate hidden states, the difference between the first and second key cache states, and the difference between the first and second value cache states output by the same network layer under the same operator kernel; and, based on each difference value and a preset difference threshold, determines whether there is an accuracy anomaly in the operator set corresponding to each network layer when the large language model calls each operator kernel to perform operations. This solves the problem in existing technologies where it is difficult to accurately verify the operator accuracy of large language models deployed on heterogeneous hardware due to differences in architecture, operator output accuracy deviation, different batch processing methods, or differences in key-value cache management mechanisms. It can accurately verify the operator accuracy of large language models deployed on heterogeneous hardware.

[0010] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0011] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0012] Figure 1 This is a flowchart of a method for verifying the accuracy of large language model operators for heterogeneous hardware, provided in Embodiment 1 of the present invention.

[0013] Figure 2 This is a flowchart of another method for verifying the precision of large language model operators for heterogeneous hardware, provided by Embodiment 2 of the present invention.

[0014] Figure 3 This is a schematic diagram of two model inference paths provided in Embodiment 2 of the present invention.

[0015] Figure 4 This is a schematic diagram of two single-network layer inference paths provided in Embodiment 2 of the present invention.

[0016] Figure 5 This is a schematic diagram of the structure of a large language model operator precision verification device for heterogeneous hardware according to Embodiment 3 of the present invention.

[0017] Figure 6 This is a schematic diagram of the structure of an electronic device provided in Embodiment 4 of the present invention. Detailed Implementation

[0018] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0019] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0020] Example 1 Figure 1 This is a flowchart of a method for verifying the accuracy of large language model operators for heterogeneous hardware, provided in Embodiment 1 of the present invention.

[0021] This embodiment is applicable to the verification of operator precision of large language models deployed on heterogeneous hardware. The method can be executed by a large language model operator precision verification device for heterogeneous hardware. This device can be implemented in hardware and / or software and can be configured in an electronic device. The electronic device can be the heterogeneous hardware where the large language model is located, or it can be a hardware control device that can control the heterogeneous hardware, as long as it can execute the large language model operator precision verification method for heterogeneous hardware. This embodiment of the invention does not limit the specific device type of the electronic device.

[0022] like Figure 1 As shown in the figure, this embodiment discloses a method for verifying the precision of large language model operators for heterogeneous hardware, including steps S110-S140.

[0023] S110. After deploying the large language model on heterogeneous hardware, obtain the first intermediate hidden state, the first key cache state, and the first value cache state output by each network layer when the large language model calls each operator kernel to perform autoregressive inference; different operator kernels correspond to different hardware instructions.

[0024] In this embodiment, heterogeneous hardware can be understood as artificial intelligence acceleration hardware that is different from GPU and CPU, such as Neural Network Processing Unit (NPU), Application Specific Integrated Circuit (ASIC), and Field Programmable Gate Array (FPGA).

[0025] Large language models, also known as big language models, refer to deep learning models trained on massive amounts of relevant data (such as text, image, or combined text-image data) capable of processing a sequence (such as text, images, or combined text-image sequences). These models can generate natural language text or understand the meaning of language text, and typically have billions of parameters. Large language models can handle various natural language tasks, such as text classification, question answering, and dialogue, and have wide applications. The input to a large language model is data, such as text, image, or combined text-image data. The large language model encodes the input data to obtain corresponding word vectors, and then decodes these word vectors to automatically process the input data and obtain the corresponding output data. For example, when the large language model is for text, text can be input, and the model processes and predicts the input text, outputting the corresponding response text; when the large language model is for images, an image can be input, and the model processes and predicts the input image, outputting the corresponding response text.

[0026] An operator kernel can be understood as the underlying implementation of an operator on heterogeneous hardware. An operator can be understood as a fundamental operator in a large language model, such as matrix multiplication and layer normalization; one operator can correspond to multiple operator kernels. Different operator kernels are implemented using different hardware instructions. Autoregressive inference can be understood as a reasoning method that iteratively generates text word by word.

[0027] In this step, specifically, after loading the weights of the large language model onto the heterogeneous hardware, the large language model can be controlled to perform autoregressive inference according to the correspondence between each preset iteration step and each operator kernel, so as to obtain the first intermediate hidden state, the first key cache state and the first value cache state output by each network layer when the large language model calls each operator kernel to perform autoregressive inference.

[0028] S120. Obtain the second intermediate hidden state, second key cache state, and second value cache state output by each network layer when the large language model calls each operator kernel to perform pre-filled inference.

[0029] In this embodiment, the pre-filled inference method can be understood as performing a one-time parallel forward computation on all tokens of the input sequence to obtain the second intermediate hidden state, the second key cache state, and the second value cache state output by each network layer.

[0030] In this step, specifically, based on the correspondence between each preset iteration step and each operator kernel, the large language model can be controlled to perform pre-filling inference according to each preset iteration step, so as to obtain the second intermediate hidden state, the second key cache state and the second value cache state output by each network layer when the large language model calls each operator kernel to perform pre-filling inference.

[0031] It is worth noting that during the operator precision verification process, the first intermediate hidden state, the first key cache state, and the first value cache state can be collected first, or the second intermediate hidden state, the second key cache state, and the second value cache state can be collected first. Alternatively, two identical heterogeneous hardware devices can be used to collect the first intermediate hidden state, the first key cache state, the first value cache state, the second intermediate hidden state, the second key cache state, and the second value cache state simultaneously. This embodiment does not limit this.

[0032] S130. Calculate the difference between the first intermediate hidden state and the second intermediate hidden state, the difference between the first key cache state and the second key cache state, and the difference between the first value cache state and the second value cache state output by the same network layer under the same operator kernel.

[0033] In this embodiment, the difference value is the cosine similarity value.

[0034] Specifically, in this step, we can calculate the cosine similarity between the first and second intermediate hidden states, the cosine similarity between the first and second key cache states, and the cosine similarity between the first and second value cache states, all within the same network layer output under the same operator kernel. Then, we can use the cosine similarity between the first and second intermediate hidden states as the difference between them; the cosine similarity between the first and second key cache states as the difference between them; and the cosine similarity between the first and second value cache states as the difference between them.

[0035] S140. Based on each difference value and a preset difference threshold, determine whether there is an accuracy anomaly in the operator set corresponding to each network layer when the large language model calls each operator kernel to perform operations.

[0036] In this step, specifically, the following operator precision anomaly verification operation can be performed for each network layer: If the difference between the first intermediate hidden state and the second intermediate hidden state is less than a preset difference threshold, or the difference between the first key cache state and the second key cache state is less than a preset difference threshold, or the difference between the first value cache state and the second value cache state is less than a preset difference threshold, then it is determined that the operator set corresponding to that network layer has a precision anomaly. If the difference between the first intermediate hidden state and the second intermediate hidden state is greater than or equal to a preset difference threshold, the difference between the first key cache state and the second key cache state is greater than or equal to a preset difference threshold, and the difference between the first value cache state and the second value cache state is greater than or equal to a preset difference threshold, then it is determined that the operator set corresponding to that network layer does not have a precision anomaly.

[0037] The advantage of this setup is that by comparing the differences in intermediate states between autoregressive decoding and pre-filling modes, the computational equivalence in the reasoning process of large language models can be fully utilized without relying on external hardware, thereby achieving automated verification of the accuracy of large language model operators on heterogeneous hardware and reducing the cost of verifying large language model operators on heterogeneous hardware.

[0038] Optionally, before determining whether there are precision anomalies in the operator set corresponding to each network layer when the large language model calls each operator kernel to perform operations based on each difference value and a preset difference threshold, the method further includes: obtaining the length of the first iteration input sequence and the operation data format when the large language model calls each operator kernel to perform pre-filled inference and autoregressive inference respectively; and determining the preset difference threshold based on the length of the first iteration input sequence, the operation data format, and the preset safety coefficient.

[0039] Specifically, because addition does not satisfy the associative law in finite-precision floating-point numerical representation, the sequence in the autoregressive path is accumulated sequentially word by word, while the entire sequence in the pre-filled path is calculated in parallel through matrix operations. The different floating-point accumulation orders introduce rounding errors, even if mathematically equivalent, leading to a slight deflection in the output vector direction. Therefore, to eliminate the theoretical error caused by the different floating-point calculation order, we can... The upper bound of the cumulative rounding error of sub-floating-point addition is: , For machine precision, accumulated rounding errors can cause slight changes in vector direction, reflected in a cosine similarity value slightly less than 1. This floating-point error analysis theory is used to construct... This dynamic similarity threshold function is used to determine a preset difference threshold. Among other things, Indicates the preset difference threshold; Indicates the preset safety factor; Indicates the data format for the operation; express The machine precision is as follows, when When it is a half-precision floating point ,when When it is a 16-bit brain floating point ,when When it is a single-precision floating point ; This indicates the length of the input sequence in the first iteration.

[0040] For example, assuming the large language model uses half-precision floating-point inference, the length of the input sequence in the first iteration is 128, and the preset safety factor is 3, then the preset difference threshold can be calculated to be 0.625.

[0041] The advantage of this setting is that, since the preset difference threshold decreases as the length of the input sequence in the first iteration increases, the lower limit of the allowed cosine similarity widens as the sequence length increases. This reflects the trend that the direction deviation naturally increases due to the accumulated rounding error in longer sequences, avoiding the situation where the preset difference threshold is too lenient for short sequences and too strict for long sequences, thus ensuring the effectiveness of the preset difference threshold.

[0042] It is worth noting that, in order to ensure the accuracy of the model anomaly verification results, the preset difference threshold in the technical solution of this embodiment can be used as a theoretical lower bound, and calibrated according to the model size and hardware characteristics in engineering practice. For example, in actual engineering deployments, for scenarios with high accuracy requirements, a more stringent empirical threshold can also be used.

[0043] The technical solution of this embodiment, after deploying a large language model on heterogeneous hardware, obtains the first intermediate hidden state, the first key cache state, and the first value cache state output by each network layer when the large language model calls each operator kernel to perform autoregressive inference; obtains the second intermediate hidden state, the second key cache state, and the second value cache state output by each network layer when the large language model calls each operator kernel to perform pre-filled inference; calculates the difference between the first and second intermediate hidden states, the difference between the first and second key cache states, and the difference between the first and second value cache states output by the same network layer under the same operator kernel; and, based on each difference value and a preset difference threshold, determines whether there is an accuracy anomaly in the operator set corresponding to each network layer when the large language model calls each operator kernel to perform operations. This solves the problem in existing technologies where it is difficult to accurately verify the operator accuracy of large language models deployed on heterogeneous hardware due to differences in architecture, operator output accuracy deviation, different batch processing methods, or differences in key-value cache management mechanisms. It can accurately verify the operator accuracy of large language models deployed on heterogeneous hardware.

[0044] Example 2 Figure 2This is a flowchart of another method for verifying the precision of large language model operators for heterogeneous hardware according to Embodiment 2 of the present invention. This embodiment is a further optimization and extension based on the above embodiments and can be combined with various optional technical solutions in the above embodiments.

[0045] like Figure 2 As shown in the figure, this embodiment discloses a method for verifying the precision of large language model operators for heterogeneous hardware, including steps S210-S280.

[0046] S210. After deploying the large language model on heterogeneous hardware, obtain the first intermediate hidden state, the first key cache state, and the first value cache state output by each network layer when the large language model calls each operator kernel to perform autoregressive inference; different operator kernels correspond to different hardware instructions.

[0047] Optionally, after deploying the large language model on heterogeneous hardware, the method further includes: setting the large language model to deterministic reasoning mode, disabling the dropout layer in the large language model, and setting the sampling strategy of the large language model to a greedy search strategy or a fixed random seed strategy.

[0048] Among them, the greedy search strategy, by selecting the term with the highest probability as the output at each step of the model decoding stage, can eliminate the interference of random sampling perturbations on the operator accuracy comparison. The fixed random seed strategy, by configuring a fixed random seed during the model inference process, can eliminate the random fluctuations introduced by the sampling mechanism and ensure the reproducibility of the inference results.

[0049] The advantage of this setup is that by setting the large language model to deterministic reasoning mode, disabling the dropout layer in the large language model, and setting the sampling strategy of the large language model to a greedy search strategy or a fixed random seed strategy, we can ensure that the reasoning process is free from random interference.

[0050] Optionally, when the large language model calls each operator kernel to perform autoregressive inference, the first intermediate hidden state, the first key cache state, and the first value cache state output by each network layer are obtained, including: obtaining each preset iteration step number and the correspondence between each preset iteration step number and each operator kernel; according to the correspondence, controlling the large language model to perform autoregressive inference according to each preset iteration step number, so as to obtain the first intermediate hidden state, the first key cache state, and the first value cache state output by each network layer when the large language model calls each operator kernel to perform autoregressive inference.

[0051] The preset iteration steps can be understood as the total number of iteration steps pre-configured before generating text; in autoregressive inference, each iteration step generates a word, as well as the first intermediate hidden state, the first key cache state, and the first value cache state corresponding to that word.

[0052] Specifically, in the implementation of operator libraries on heterogeneous hardware, the same operator, such as matrix multiplication and attention mechanism, will select different operator kernel implementation paths based on the length of the input sequence of the final iteration. For example, when the length of the input sequence of the final iteration is less than or equal to 64, a register-level optimized small matrix operator kernel is preferred; when the length of the input sequence of the final iteration is greater than or equal to 65 but less than or equal to 512, a shared memory optimized medium matrix operator kernel is preferred; when the length of the input sequence of the final iteration is greater than 512, a block-based computation operator kernel is preferred. Therefore, when the length of the input sequence of the final iteration is equal to the sum of the length of the input sequence of the first iteration and the preset number of iterations, the large language model can be controlled to perform autoregressive inference according to each preset number of iterations to obtain the first intermediate hidden state, the first key cache state, and the first value cache state output by each network layer when the large language model calls different operator kernels to perform autoregressive inference.

[0053] In one example, assuming the preset iteration steps are 1, 8, 16, 32, 54, 64, 65, 100, 128, 256, 502, 512, 513, and 1024, and these values ​​cover the boundary points of operator kernel switching (e.g., 65 and 513 are critical values ​​for operator kernel switching), then with a preset iteration step count of 5 and a first-round iteration input sequence length of 10, the large language model can be controlled to call the small matrix operator kernel to perform autoregressive inference, in order to obtain the results from the network layers in the large language model at time steps. The output includes the first intermediate hidden state, the first key cache state, and the first value cache state. Specifically, such as... Figure 3 As shown, in the autoregressive inference path, the large language model starts from the initial input sequence Begin by performing autoregressive inference step by step: at each time step ,enter After the first network layer, the second network layer, and so on... The computation proceeds layer by layer through each network layer, with each layer outputting its corresponding intermediate hidden state, key cache state, and value cache state; at each time step... The input tokens generated in the previous time step are processed through all layers, and the intermediate hidden states, key cache states, and value cache states of each layer are updated accordingly. This process is repeated for multiple iterations until the next time step. Ultimately, the intermediate hidden states, key cache states, and value cache states of each layer are obtained. Specifically, at each time step... The first network layer, the second network layer, and the third network layer The intermediate hidden states output by each network layer can be denoted as follows: , and The key cache state can be recorded as follows: , and The value cache state can be recorded as follows: , and At time step The first network layer, the second network layer, and the third network layer The intermediate hidden states output by each network layer can be denoted as follows: , and The key cache state can be recorded as follows: , and The value cache state can be recorded as follows: , and At time step The first network layer, the second network layer, and the third network layer The intermediate hidden states output by each network layer can be denoted as follows: , and The key cache state can be recorded as follows: , and The value cache state can be recorded as follows: , and .

[0054] Furthermore, since the preset iteration number is 55 and the length of the input sequence in the first iteration is 10, the length of the input sequence in the last iteration is 65. At this point, the operator kernel switching can be triggered, thereby controlling the large language model to call the medium matrix operator kernel to perform autoregressive inference, so as to obtain the results of each network layer in the large language model at time steps. The output includes the first intermediate hidden state, the first key cache state, and the first value cache state. Similarly, given a preset iteration count of 503 and an initial iteration input sequence length of 10, the final iteration input sequence length is 513. This triggers an operator kernel switch, controlling the large language model to call the block-based computation operator kernel to execute autoregressive inference, thus obtaining the results from each network layer in the large language model at time steps. The output includes the first intermediate hidden state, the first key cache state, and the first value cache state.

[0055] The advantage of this setup is that, compared to existing technologies that struggle to cover the dynamically changing operator kernels in autoregressive inference of large language models, leading to an inability to determine operator precision under different operator kernels, the technical solution in this embodiment can quickly and accurately determine operator precision under different operator kernels by adjusting the preset number of iterations to switch operator kernels.

[0056] S220. When the large language model calls each operator kernel to perform pre-filled inference, the second intermediate hidden state, the second key cache state, and the second value cache state output by each network layer are obtained.

[0057] In this step, specifically, the large language model can be controlled to perform pre-filling inference according to each preset iteration step, so as to obtain the second intermediate hidden state, the second key cache state and the second value cache state output by each network layer when the large language model calls each operator kernel to perform pre-filling inference.

[0058] For example, assuming a preset iteration count of 5, the large language model can be controlled to perform pre-padded inference at this preset iteration count of 5, so as to obtain the second intermediate hidden state, the second key cache state, and the second value cache state output by each network layer when the large language model calls each operator kernel to perform pre-padded inference. Specifically, such as Figure 3 As shown, the complete sequence can be included in the pre-filled inference path. A large language model is input at once, passing through the first network layer, the second network layer, and so on up to the third. The computation of each network layer is performed, and the last one is extracted. Location, that is The corresponding positions represent the intermediate hidden states, key cache states, and value cache states output by each network layer. Among these, the pre-filled inference path... At the corresponding positions, the first network layer, the second network layer, and the third network layer The intermediate hidden states output by each network layer can be denoted as follows: , and The key cache state can be recorded as follows: , and The value cache state can be recorded as follows: , and .

[0059] S230. Calculate the difference between the first intermediate hidden state and the second intermediate hidden state, the difference between the first key cache state and the second key cache state, and the difference between the first value cache state and the second value cache state output by the same network layer under the same operator kernel.

[0060] In this embodiment, the difference is the cosine similarity, which ranges from -1 to 1. A cosine similarity closer to 1 indicates that the two vectors are more aligned in direction, meaning the output accuracy of the two inference paths at this network layer is higher. A cosine similarity deviating from 1 indicates an accuracy deviation. The dimension of the intermediate hidden states is [1, D], where D is a predefined dimension of the hidden layer.

[0061] In this step, specifically, the difference between the first and second intermediate hidden states output by the same network layer under the same operator kernel can be determined using the following specific calculation formula: .

[0062] in, Indicates the first The difference between the first and second intermediate hidden states output by each network layer Indicates the first The first intermediate hidden state output by each network layer Indicates the first The second intermediate hidden state output by each network layer The Euclidean norm of the first intermediate hidden state is given. The Euclidean norm representing the second intermediate hidden state. The first intermediate hidden state represents the... scalar elements The second intermediate hidden state represents the first scalar elements Indicates the index of the hidden layer dimension.

[0063] S240. Based on each difference value and a preset difference threshold, determine whether there are any precision anomalies in the operator set corresponding to each network layer when the large language model calls each operator kernel to perform operations.

[0064] For example, assuming the preset difference threshold is 0.9995, if the large language model calls the small matrix operator kernel, the cosine similarity values ​​of all network layers are higher than the preset difference threshold. However, when the large language model calls the medium matrix operator kernel, the cosine similarity value of the 8th network layer drops sharply to 0.72, and the cosine similarity value of the 15th network layer drops sharply to 0.81. This results in the 8th and 15th network layers having cosine similarity values ​​lower than the preset difference threshold. In this case, it is determined that the operator sets corresponding to the 8th and 15th network layers have abnormal precision when the large language model calls the medium matrix operator kernel.

[0065] Furthermore, to distinguish whether the error arises during forward computation or is introduced during cache writing, the comparison results between each difference value and a preset difference threshold can be determined simultaneously. Taking the 8th network layer as an example, assuming that the difference between the first and second intermediate hidden states output by this layer is greater than or equal to the preset difference threshold, but the difference between the first and second key cache states output by this layer is less than the preset difference threshold, and the difference between the first and second value cache states output by this layer is also less than the preset difference threshold, then it is determined that the error in this layer is introduced during cache writing, and the key-value cache writing operator of this layer has a precision problem.

[0066] S250. If an abnormal network layer with an abnormal precision occurs when the large language model calls each operator kernel to perform operations, then obtain the abnormal operator kernel corresponding to the abnormal network layer, as well as each sub-module within the abnormal network layer.

[0067] Taking a large language model using a Transformer architecture as an example, since each network layer of this large language model consists of multiple sub-modules, and the typical structure is "input → first layer normalization sub-module → attention mechanism sub-module (including query key-value projection, score calculation, activation function and output projection) → first residual connection sub-module → second layer normalization sub-module → feedforward network sub-module (gated projection, up-dimensional projection, activation function and down-dimensional projection) → second residual connection sub-module → output", when it is determined that there is an accuracy anomaly in a certain layer of the model, the abnormal operator kernel corresponding to the abnormal network layer, as well as each sub-module in the abnormal network layer, can be obtained to locate the abnormal root factor module.

[0068] S260. Insert a data recording hook at the output of each submodule, and obtain the intermediate state of autoregressive inference output by each submodule when the large language model calls the exception operator kernel to perform autoregressive inference, and the intermediate state of pre-filled inference output by each submodule when the large language model calls the exception operator kernel to perform pre-filled inference.

[0069] For example, such as Figure 4 As shown, assuming the anomaly detection network layer is the 12th network layer of the large language model, the preset iteration step count is 5, and each network layer in the large language model contains, from top to bottom, a feedforward network submodule, an attention mechanism submodule, and a layer normalization submodule, then in the autoregressive inference path, the large language model performs autoregressive inference step by step: at time step ,enter After sub-module calculations by the feedforward network sub-module, attention mechanism sub-module, and layer normalization sub-module, the intermediate hidden state, key cache state, and value cache state corresponding to the output of each sub-module are obtained; at time step The input is the intermediate hidden state generated by the previous network layer at the same time step. Similarly, all sub-modules calculate and update their intermediate hidden states, key cache states, and value cache states; this process continues through multiple iterations until the next time step. Ultimately, the intermediate hidden states, key cache states, and value cache states of each submodule are obtained. In the pre-filled inference path, A large language model is input at once, and after computation by the feedforward network submodule, attention mechanism submodule, and layer normalization submodule, the last one is extracted. Location, that is The corresponding positions output the intermediate hidden state, key cache state, and value cache state of each submodule. Specifically, at each time step... The input to the feedforward network submodule Indicates time step The intermediate hidden states output by the previous network layer, the intermediate hidden states output by the feedforward network submodule, the attention mechanism submodule, and the layer normalization submodule can be denoted as follows: , and The key cache state can be recorded as follows: , and The value cache state can be recorded as follows: , and At time step The input to the feedforward network submodule Indicates time step The intermediate hidden states output by the previous network layer, the intermediate hidden states output by the feedforward network submodule, the attention mechanism submodule, and the layer normalization submodule can be denoted as follows: , and The key cache state can be recorded as follows: , and The value cache state can be recorded as follows: , and At time step The input to the feedforward network submodule This represents the intermediate hidden state output by the previous network layer at this time step. The intermediate hidden states output by the feedforward network submodule, the attention mechanism submodule, and the layer normalization submodule can be denoted as follows: , and The key cache state can be recorded as follows: , and The value cache state can be recorded as follows: , and Pre-filled inference paths at time steps The input to the feedforward network submodule This indicates the time step. The intermediate hidden states output by the previous network layer, the intermediate hidden states output by the feedforward network submodule, the attention mechanism submodule, and the layer normalization submodule can be denoted as follows: , and The key cache state can be recorded as follows: , and The value cache state can be recorded as follows: , and .

[0070] S270. Based on the autoregressive inference intermediate states and pre-filled inference intermediate states corresponding to each sub-module, determine the abnormal root factor module in each sub-module.

[0071] In this embodiment, the autoregressive inference intermediate states include the autoregressive intermediate hidden state, the autoregressive key cache state, and the autoregressive value cache state, and the pre-filled inference intermediate states include the pre-filled intermediate hidden state, the pre-filled key cache state, and the pre-filled value cache state.

[0072] Specifically, in this step, the first state difference value between the autoregressive intermediate hidden state and the pre-filled intermediate hidden state, the second state difference value between the autoregressive key cache state and the pre-filled key cache state, and the third state difference value between the autoregressive value cache state and the pre-filled value cache state for each sub-module can be calculated. Then, the first sub-module whose corresponding arbitrary state difference value is greater than a preset state difference threshold is designated as the abnormal root factor module.

[0073] Optionally, based on the autoregressive inference intermediate states and pre-filled inference intermediate states corresponding to each sub-module, an anomaly root factor module is determined in each sub-module. This includes: calculating, according to the inverse operation order of each sub-module, the first state difference value between the autoregressive intermediate hidden state and the pre-filled intermediate hidden state, the second state difference value between the autoregressive key cache state and the pre-filled key cache state, and the third state difference value between the autoregressive value cache state and the pre-filled value cache state; and designating the first sub-module whose corresponding arbitrary state difference value is greater than a preset state difference threshold as the anomaly root factor module. The state difference value is the cosine similarity value.

[0074] Specifically, following a backward-to-forward approach, the system checks the first state difference between the autoregressive intermediate hidden state and the pre-filled intermediate hidden state, the second state difference between the autoregressive key cache state and the pre-filled key cache state, and the third state difference between the autoregressive value cache state and the pre-filled value cache state for each submodule. Then, the first submodule whose corresponding state difference value exceeds a preset state difference threshold is designated as the root factor module for anomalies.

[0075] Taking a large language model employing a transformer architecture as an example, the cosine similarity value of each submodule in the anomalous network layer can be checked sequentially in the following order: "second residual connection submodule → feedforward network submodule → second layer normalization submodule → first residual connection submodule → attention mechanism submodule → first layer normalization submodule". When, starting from the first layer normalization submodule, any state difference value of the attention mechanism submodule is found to be greater than a preset state difference threshold, the attention mechanism submodule is determined to be an anomalous root factor module.

[0076] S280. Determine the abnormal operators based on the abnormal root factor module, and formulate the abnormal repair scheme corresponding to the abnormal operators.

[0077] In this step, specifically, a target state difference value greater than a preset state difference threshold can be obtained from the state difference values ​​corresponding to the anomaly root factor module. Then, based on the target state difference value, an anomaly operator can be determined from the operators corresponding to the anomaly root factor module. Finally, based on a preset knowledge base, an anomaly repair plan corresponding to the anomaly operator can be formulated.

[0078] For example, the operators contained in the abnormal root factor module can be analyzed. If the abnormal operator is determined to be the attention score calculation operator, then when the abnormal root cause of the attention score calculation operator is that the matrix multiplication operator uses a specific block strategy and the accumulator precision inside the block is insufficient, that is, it uses half-precision floating-point accumulation instead of single-precision floating-point accumulation, which leads to an unexpected directional deviation in the query key-value matrix multiplication, the following abnormal repair scheme is formulated based on the preset knowledge base: upgrade the accumulator of the abnormal operator kernel to single-precision floating-point precision, and re-execute the verification after repair until the cosine similarity values ​​of each abnormal network layer are greater than or equal to the preset difference threshold.

[0079] The technical solution of this embodiment, when a large language model calls each operator kernel to perform operations and an abnormal network layer with an abnormal precision in the corresponding operator set appears, obtains the abnormal operator kernel corresponding to the abnormal network layer, as well as each sub-module within the abnormal network layer; inserts a data recording hook at the output end of each sub-module, and obtains the intermediate state of autoregressive inference output by each sub-module when the large language model calls the abnormal operator kernel to perform autoregressive inference, and the intermediate state of pre-filled inference output by each sub-module when the large language model calls the abnormal operator kernel to perform pre-filled inference; based on the intermediate states of autoregressive inference and pre-filled inference corresponding to each sub-module, an abnormal root factor module is determined in each sub-module; based on the abnormal root factor module, an abnormal operator is determined, and an abnormal repair scheme corresponding to the abnormal operator is formulated. This technical means can achieve accurate verification and fault location of the precision of large language model operators deployed on heterogeneous hardware without relying on external hardware, thereby reducing the debugging cost of large language model operators on heterogeneous hardware, suppressing the accumulation of floating-point errors, and ensuring the consistency of model inference precision and output. Furthermore, by accurately verifying the operator precision of the large language model deployed on heterogeneous hardware, implicit design flaws at the underlying level of heterogeneous hardware can be efficiently captured.

[0080] Example 3 Figure 5 This is a schematic diagram of the structure of a large language model operator precision verification device for heterogeneous hardware according to Embodiment 3 of the present invention.

[0081] This embodiment is applicable to the verification of operator precision of large language models deployed on heterogeneous hardware. The large language model operator precision verification device for heterogeneous hardware can be implemented in hardware and / or software. The large language model operator precision verification device for heterogeneous hardware can be configured in an electronic device. The electronic device can be the heterogeneous hardware where the large language model is located, or it can be a hardware control device that can control the above-mentioned heterogeneous hardware, as long as it can execute the large language model operator precision verification method for heterogeneous hardware. This embodiment of the invention does not limit the specific device type of the electronic device.

[0082] like Figure 5 As shown, the large language model operator precision verification device for heterogeneous hardware disclosed in this embodiment includes a first state acquisition module 51, a second state acquisition module 52, a difference value determination module 53, and an anomaly recognition module 54.

[0083] The first state acquisition module 51 is used to acquire the first intermediate hidden state, the first key cache state, and the first value cache state output by each network layer when the large language model calls each operator kernel to perform autoregressive inference after the large language model is deployed on heterogeneous hardware; different operator kernels correspond to different hardware instructions.

[0084] The second state acquisition module 52 is used to acquire the second intermediate hidden state, the second key cache state, and the second value cache state output by each network layer when the large language model calls each operator kernel to perform pre-filled inference.

[0085] The difference value determination module 53 is used to calculate the difference value between the first intermediate hidden state and the second intermediate hidden state, the difference value between the first key cache state and the second key cache state, and the difference value between the first value cache state and the second value cache state output by the same network layer under the same operator kernel.

[0086] The anomaly identification module 54 is used to determine whether there are precision anomalies in the operator set corresponding to each network layer when the large language model calls each operator kernel to perform operations, based on each difference value and a preset difference threshold.

[0087] The technical solution in this embodiment, through the cooperation of the first state acquisition module 51, the second state acquisition module 52, the difference value determination module 53, and the anomaly identification module 54, solves the problem that existing technologies are unable to accurately verify the operator precision of large language models deployed on heterogeneous hardware due to differences in architecture, operator output precision deviation, different batch processing methods, or differences in key-value cache management mechanisms. It can accurately verify the operator precision of large language models deployed on heterogeneous hardware.

[0088] Optionally, the device further includes an anomaly repair module, which is used to: if an abnormal network layer with an abnormal precision exists in the corresponding operator set when the large language model calls each operator kernel to perform operations, then obtain the abnormal operator kernel corresponding to the abnormal network layer, as well as each sub-module within the abnormal network layer; insert a data recording hook at the output end of each sub-module, and obtain the intermediate state of autoregressive inference output by each sub-module when the large language model calls the abnormal operator kernel to perform autoregressive inference, and the intermediate state of pre-filled inference output by each sub-module when the large language model calls the abnormal operator kernel to perform pre-filled inference; determine the abnormal root factor module in each sub-module based on the intermediate state of autoregressive inference and the intermediate state of pre-filled inference corresponding to each sub-module respectively; determine the abnormal operator based on the abnormal root factor module, and formulate an anomaly repair scheme corresponding to the abnormal operator.

[0089] Optionally, when the autoregressive inference intermediate states include autoregressive intermediate hidden states, autoregressive key cache states, and autoregressive value cache states, and the pre-filled inference intermediate states include pre-filled intermediate hidden states, pre-filled key cache states, and pre-filled value cache states, the anomaly repair module is specifically used to: calculate, according to the inverse operation order of each sub-module, the first state difference value between the autoregressive intermediate hidden state and the pre-filled intermediate hidden state, the second state difference value between the autoregressive key cache state and the pre-filled key cache state, and the third state difference value between the autoregressive value cache state and the pre-filled value cache state, respectively; and designate the first sub-module whose corresponding arbitrary state difference value is greater than a preset state difference threshold as the anomaly root factor module.

[0090] Optionally, the difference value is the cosine similarity value.

[0091] Optionally, the device further includes a threshold determination module, which is used to: obtain the input sequence length and operation data format when the large language model calls each operator kernel to perform pre-filled inference and autoregressive inference respectively; and determine a preset difference threshold based on the input sequence length, operation data format and preset safety coefficient.

[0092] Optionally, the first state acquisition module 51 is specifically used to: acquire each preset iteration step number, and the correspondence between each preset iteration step number and each operator kernel; according to the correspondence, control the large language model to perform autoregressive inference according to each preset iteration step number, so as to obtain the first intermediate hidden state, the first key cache state and the first value cache state output by each network layer when the large language model calls each operator kernel to perform autoregressive inference.

[0093] Optionally, the device also includes a model configuration module, which is used to: set the large language model to deterministic reasoning mode, turn off the dropout layer in the large language model, and set the sampling strategy of the large language model to a greedy search strategy or a fixed random seed strategy.

[0094] The large language model operator precision verification device for heterogeneous hardware provided in this embodiment of the invention can execute the large language model operator precision verification method for heterogeneous hardware provided in any embodiment of the invention, and has the corresponding functional modules and beneficial effects of the method execution. Content not described in detail in this embodiment can be referred to the description in any method embodiment of this application.

[0095] Example 4 Figure 6 A schematic diagram of the structure of an electronic device 10 that can be used to implement embodiments of the present invention is shown. For example... Figure 6As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory 12 or a random access memory 13, communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer program stored in the read-only memory 12 or loaded from storage unit 18 into the random access memory 13. The random access memory 13 may also store various programs and data required for the operation of the electronic device 10. The processor 11, read-only memory 12, and random access memory 13 are interconnected via a bus 14. An input / output interface 15 is also connected to the bus 14.

[0096] Multiple components in electronic device 10 are connected to input / output interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of monitors, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0097] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as the large language model operator precision verification method for heterogeneous hardware.

[0098] In some embodiments, the large language model operator precision verification method for heterogeneous hardware can be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program can be loaded and / or installed on electronic device 10 via read-only memory 12 and / or communication unit 19. When the computer program is loaded into random access memory 13 and executed by processor 11, one or more steps of the large language model operator precision verification method for heterogeneous hardware described above can be performed. Alternatively, in other embodiments, processor 11 can be configured to perform the large language model operator precision verification method for heterogeneous hardware by any other suitable means (e.g., by means of firmware).

[0099] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0100] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0101] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory, read-only memory, erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0102] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0103] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or middleware components (e.g., application servers), or frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0104] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0105] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0106] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A method for verifying the precision of large language model operators for heterogeneous hardware, characterized in that, The method for verifying the accuracy of large language model operators for heterogeneous hardware includes: After deploying the large language model on heterogeneous hardware, the first intermediate hidden state, the first key cache state, and the first value cache state output by each network layer are obtained when the large language model calls each operator kernel to perform autoregressive inference; different operator kernels correspond to different hardware instructions; When the large language model calls each operator kernel to perform pre-filled inference, the second intermediate hidden state, the second key cache state, and the second value cache state output by each network layer are obtained. Calculate the difference between the first intermediate hidden state and the second intermediate hidden state, the difference between the first key cache state and the second key cache state, and the difference between the first value cache state and the second value cache state in the same network layer output under the same operator kernel; Based on the aforementioned difference values ​​and preset difference thresholds, it is determined whether there are precision anomalies in the operator sets corresponding to each network layer when the large language model calls each of the aforementioned operator kernels to perform operations.

2. The method for verifying the accuracy of large language model operators for heterogeneous hardware according to claim 1, characterized in that, After determining whether there are precision anomalies in the operator set corresponding to each network layer when the large language model calls each operator kernel to perform operations based on the aforementioned difference values ​​and preset difference thresholds, the method further includes: If, when the large language model calls each of the operator kernels to perform operations, an abnormal network layer with an abnormal precision exists in the corresponding operator set, then the abnormal operator kernel corresponding to the abnormal network layer, as well as each sub-module within the abnormal network layer, are obtained. Insert a data recording hook at the output of each submodule, and obtain the intermediate state of autoregressive inference output by each submodule when the large language model calls the abnormal operator kernel to perform autoregressive inference, and the intermediate state of pre-filled inference output by each submodule when the large language model calls the abnormal operator kernel to perform pre-filled inference. Based on the autoregressive inference intermediate state and pre-filled inference intermediate state corresponding to each of the sub-modules, an abnormal root factor module is determined in each of the sub-modules. Based on the aforementioned abnormal root factor module, abnormal operators are determined, and abnormal repair schemes corresponding to the abnormal operators are formulated.

3. The method for verifying the accuracy of large language model operators for heterogeneous hardware according to claim 2, characterized in that, The autoregressive inference intermediate states include autoregressive intermediate hidden states, autoregressive key cache states, and autoregressive value cache states; the pre-filled inference intermediate states include pre-filled intermediate hidden states, pre-filled key cache states, and pre-filled value cache states. Based on the autoregressive inference intermediate states and pre-filled inference intermediate states corresponding to each of the sub-modules, an anomaly root factor module is determined in each of the sub-modules, including: According to the inverse operation order of each sub-module, calculate the first state difference value between the autoregressive intermediate hidden state and the prefilled intermediate hidden state, the second state difference value between the autoregressive key cache state and the prefilled key cache state, and the third state difference value between the autoregressive value cache state and the prefilled value cache state for each sub-module. The first submodule whose corresponding arbitrary state difference value is greater than the preset state difference threshold is designated as the abnormal root factor module.

4. The method for verifying the accuracy of large language model operators for heterogeneous hardware according to claim 1, characterized in that, The difference value is the cosine similarity value.

5. The method for verifying the accuracy of large language model operators for heterogeneous hardware according to claim 1, characterized in that, Before determining whether there are precision anomalies in the operator set corresponding to each network layer when the large language model calls each operator kernel to perform operations based on the aforementioned difference values ​​and preset difference thresholds, the method further includes: Obtain the length of the first iteration input sequence and the data format of the operation when the large language model calls each operator kernel to perform pre-filled inference and autoregressive inference respectively; Based on the length of the input sequence in the first round of iteration, the data format of the operation, and the preset security coefficient, a preset difference threshold is determined.

6. The method for verifying the accuracy of large language model operators for heterogeneous hardware according to claim 1, characterized in that, When the large language model calls each operator kernel to perform autoregressive inference, the first intermediate hidden state, the first key cache state, and the first value cache state output by each network layer are obtained, including: Obtain each preset iteration step number, and the correspondence between each preset iteration step number and each operator kernel; Based on the correspondence, the large language model is controlled to perform autoregressive inference according to each preset iteration step, so as to obtain the first intermediate hidden state, the first key cache state and the first value cache state output by each network layer when the large language model calls each operator kernel to perform autoregressive inference.

7. The method for verifying the precision of large language model operators for heterogeneous hardware according to claim 6, characterized in that, After deploying large language models on heterogeneous hardware, the following is also included: Set the large language model to deterministic reasoning mode, disable the dropout layer in the large language model, and set the sampling strategy of the large language model to a greedy search strategy or a fixed random seed strategy.

8. A precision verification device for large language model operators for heterogeneous hardware, characterized in that, The large language model operator accuracy verification device for heterogeneous hardware includes: The first state acquisition module is used to acquire the first intermediate hidden state, the first key cache state, and the first value cache state output by each network layer when the large language model calls each operator kernel to perform autoregressive inference after the large language model is deployed on heterogeneous hardware; different operator kernels correspond to different hardware instructions; The second state acquisition module is used to acquire the second intermediate hidden state, the second key cache state and the second value cache state output by each network layer when the large language model calls each operator kernel to perform pre-filled inference. The difference value determination module is used to calculate the difference between the first intermediate hidden state and the second intermediate hidden state, the difference between the first key cache state and the second key cache state, and the difference between the first value cache state and the second value cache state output by the same network layer under the same operator kernel. An anomaly detection module is used to determine, based on the difference values ​​and preset difference thresholds, whether there are precision anomalies in the operator set corresponding to each network layer when the large language model calls each operator kernel to perform operations.

9. An electronic device, characterized in that, The electronic device includes: At least one processor; and a memory communicatively connected to said at least one processor; The memory stores a computer program that can be executed by the at least one processor, which is then executed by the at least one processor to enable the at least one processor to perform the large language model operator precision verification method for heterogeneous hardware as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that cause a processor to execute the method for verifying the precision of large language model operators for heterogeneous hardware as described in any one of claims 1-7.

Images