A model training consistency verification method and device, and a storage medium
By performing hash calculations and block processing in the AI chip, the problem of time-consuming target tensor copying in large model training is solved, the consistency verification efficiency and training throughput are improved, and fast and accurate problem troubleshooting is achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- SHANGHAI BIREN TECH CO LTD
- Filing Date
- 2026-03-18
- Publication Date
- 2026-06-23
AI Technical Summary
During the training of large models, copying the target tensor takes a long time and consumes a lot of resources, resulting in low consistency verification efficiency and reduced training throughput.
Hash calculations are moved from the central processing unit (CPU) to an artificial intelligence chip. The target tensor is divided into multiple blocks and aggregated features are extracted. Only the hash value of the target tensor is copied to the CPU for comparison. Methods such as XOR aggregation are used to improve the efficiency of hash calculations.
It reduces the resource consumption of artificial intelligence chips, improves model training throughput and consistency verification efficiency, shortens troubleshooting time, and improves accuracy.
Smart Images
Figure CN121882308B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of artificial intelligence chip technology, and in particular to a method, device and storage medium for consistency verification of model training. Background Technology
[0002] With the development of artificial intelligence technology, the application of large-scale models is becoming increasingly widespread. Large-scale model training involves large-scale, distributed, and long-term feature point training. Large-scale typically refers to a large number of parameters and training data; distributed training usually refers to multi-machine, multi-GPU training; and long-term training typically refers to training periods lasting several weeks or months. These characteristics make large-scale model training prone to inconsistencies in training results. Therefore, consistency verification during large-scale model training is a necessary means to promptly identify problems.
[0003] In this technology, large models are trained multiple times, with each training iteration consisting of multiple steps. For each training iteration, the Graphics Processing Unit (GPU) generates multiple target tensors. These target tensors are then synchronously copied to the Central Processing Unit (CPU). The CPU performs a hash calculation on each target tensor, obtaining and storing the corresponding hash value. Finally, the CPU compares the hash values generated from the multiple training iterations to obtain a consistency verification result.
[0004] However, large models generate target tensors with a large amount of data. When the GPU copies these target tensors to the CPU, the copying process is time-consuming and resource-intensive, resulting in low efficiency of consistency verification and a decrease in training throughput. Summary of the Invention
[0005] This application provides a consistency verification method, device, and storage medium for model training, which improves the efficiency of consistency verification and increases training throughput.
[0006] On one hand, embodiments of this application provide a consistency verification method for model training, applied to artificial intelligence chips, the method comprising:
[0007] Obtain multiple target tensors generated during model training;
[0008] For each target tensor, perform the following verification operations:
[0009] The target tensor is divided into multiple blocks, and the aggregate features of each block are extracted;
[0010] The aggregation features of each block are hashed to obtain the corresponding block hash value;
[0011] Based on the obtained multiple block hash values, the target hash value of the target tensor is obtained;
[0012] The target hash value of the target tensor is sent to the central processing unit, so that the central processing unit compares each received target hash value with the corresponding reference hash value to obtain a consistency verification result.
[0013] On one hand, embodiments of this application provide a consistency verification device for model training, applied to an artificial intelligence chip, the device comprising:
[0014] The acquisition module is used to acquire multiple target tensors generated during model training.
[0015] The verification module performs the following verification operations for each target tensor:
[0016] The target tensor is divided into multiple blocks, and the aggregate features of each block are extracted;
[0017] The aggregation features of each block are hashed to obtain the corresponding block hash value;
[0018] Based on the obtained multiple block hash values, the target hash value of the target tensor is obtained;
[0019] The target hash value of the target tensor is sent to the central processing unit, so that the central processing unit compares each received target hash value with the corresponding reference hash value to obtain a consistency verification result.
[0020] Optionally, the verification module is specifically used for:
[0021] The baseline block size is obtained based on the memory bandwidth of the AI chip.
[0022] The target tensor is divided into multiple blocks according to the aforementioned base block size.
[0023] Optionally, the aggregation feature includes at least one of statistical features, location features, and seed features;
[0024] The verification module is specifically used for:
[0025] For each block, the summation result of multiple elements contained in the block and the mean of the multiple elements are determined, and the statistical characteristics of the block are obtained based on the summation result and the mean.
[0026] Based on the position index of the largest element and the position index of the smallest element in the block, the positional features of the block are obtained;
[0027] From the multi-step iterative training included in the model training process, determine the iterative training that generates the target tensor, and obtain the iteration step identifier of the iterative training;
[0028] Based on the block identifier and the iteration step identifier, the seed features of the block are obtained.
[0029] Optionally, the verification module is specifically used for:
[0030] For each block, the statistical features, positional features, and seed features of the block are weighted and fused to obtain the target fusion features;
[0031] The target fusion feature is hashed to obtain the block hash value of the block.
[0032] Optionally, the verification module is specifically used for:
[0033] The target hash value of the target tensor is obtained by XOR aggregation of the multiple block hash values.
[0034] Optionally, the verification module is specifically used for:
[0035] The multiple block hash values are XORed and aggregated to obtain the first hash value;
[0036] From the multi-step iterative training included in the model training process, determine the iterative training that generates the target tensor, and obtain the iteration step identifier of the iterative training;
[0037] Obtain the hardware identifier of the hardware unit that generates the target tensor;
[0038] The shape information of the target tensor, the iteration step identifier, the hardware identifier, and the first hash value are fused together to obtain the target hash value of the target tensor.
[0039] Optionally, the verification module is specifically used for:
[0040] The shape information of the target tensor is hashed to obtain a second hash value;
[0041] Perform a hash calculation on the iteration step identifier to obtain a third hash value;
[0042] Perform a hash calculation on the hardware identifier to obtain a fourth hash value;
[0043] The first hash value, the second hash value, the third hash value, and the fourth hash value are XORed together to obtain the target hash value of the target tensor.
[0044] Optionally, the model training process and the verification operation are executed asynchronously.
[0045] On one hand, embodiments of this application provide a computer device, including a memory, an artificial intelligence chip, and a computer program stored in the memory and running on the artificial intelligence chip. When the artificial intelligence chip executes the computer program, it implements the steps of the consistency verification method for model training described above.
[0046] On one hand, embodiments of this application provide a computer-readable storage medium storing a computer program executable by a computer device, which, when run on the computer device, causes the computer device to perform the steps of the consistency verification method for model training described above.
[0047] On one hand, embodiments of this application provide a computer program product, the computer program product including a computer program stored on a computer-readable storage medium, the computer program including program instructions, which, when executed by a computer device, cause the computer device to perform the steps of the above-described model training consistency verification method.
[0048] In this embodiment, the hash calculation operation is first transferred from the CPU to the artificial intelligence chip. In this way, the artificial intelligence chip only needs to copy the target hash value of the target tensor to the CPU, instead of copying the entire target tensor. This greatly reduces the amount of data copied, thereby reducing the resource consumption of the artificial intelligence chip and improving the model training throughput.
[0049] Secondly, the target tensor is divided into multiple blocks, and the aggregate features of each block are extracted. Then, a hash calculation is performed on the aggregate features of each block, instead of hashing the entire bytes of the target tensor. This greatly improves the efficiency of hash calculation, thereby improving the efficiency of consistency verification.
[0050] In addition, hash calculations are performed on each block of the target tensor. This allows for the further location of the biased blocks in the target tensor when inconsistent target tensors are found during consistency verification. This transforms the investigation of inconsistencies from "full recalculation" to "block location". As a result, the investigation time can be reduced from hours to minutes, which not only greatly reduces the time spent on problem investigation but also improves the accuracy of problem investigation. Attached Figure Description
[0051] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0052] Figure 1 This is a schematic diagram of the structure of an artificial intelligence chip provided in an embodiment of this application;
[0053] Figure 2 A flowchart illustrating a consistency verification method for model training provided in an embodiment of this application;
[0054] Figure 3 A flowchart illustrating another consistency verification method for model training provided in this application embodiment;
[0055] Figure 4 A schematic diagram of the structure of a consistency verification device for model training provided in an embodiment of this application;
[0056] Figure 5 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation
[0057] To make the objectives, technical solutions, and beneficial effects of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.
[0058] refer to Figure 1 This is a structural diagram of an artificial intelligence chip applicable to an embodiment of this application. The artificial intelligence chip 100 includes at least: video memory 101 and multiple computing units 102, wherein the computing units 102 may be streaming processing clusters (SPCs). The video memory 101 may be high-bandwidth memory (HBM) or other types of memory.
[0059] In the embodiments of this application, model training can be performed using at least one artificial intelligence chip 100; or model training can be performed using at least one computing unit 102 in the artificial intelligence chip 100. This application does not specifically limit the specific methods used.
[0060] For the same model (such as a large model), multiple training sessions can be performed, and the training results from multiple training sessions can be verified for consistency to ensure the consistency of the training results.
[0061] Each model training process includes multiple iterations (also known as multiple rounds of iterative training), with each iteration corresponding to an iteration step identifier (step_id). Each iteration includes: forward computation, backward gradient computation, and parameter update. During forward computation, each network layer of the model generates at least one feature tensor; during backward gradient computation, each network layer of the model generates at least one gradient tensor; and during parameter update, multiple updated weight tensors of the model are generated. The target tensor for consistency verification in this application includes at least one of the aforementioned feature tensors, gradient tensors, and weight tensors.
[0062] For each target tensor, the AI chip 100 divides the target tensor into multiple blocks and extracts the aggregation features of each block; then, it performs hash calculation on the aggregation features of each block to obtain the corresponding block hash value; based on the obtained multiple block hash values, it obtains the target hash value of the target tensor; then, it sends the target hash value of the target tensor to the CPU, so that the CPU compares each received target hash value with the corresponding reference hash value to obtain a consistency verification result.
[0063] In practical applications, after obtaining multiple target hash values for the first model training, the AI chip 100 sends each target hash value to the central processing unit. The central processing unit can use the received target hash values as reference hash values for subsequent consistency verification; of course, it can also update each reference hash value with the target hash values generated by other model training iterations, which is not specifically limited in this application.
[0064] In this embodiment, the hash calculation operation is first transferred from the CPU to the artificial intelligence chip. In this way, the artificial intelligence chip only needs to copy the target hash value of the target tensor to the CPU, instead of copying the entire target tensor. This greatly reduces the amount of data copied, thereby reducing the resource consumption of the artificial intelligence chip and improving the model training throughput.
[0065] Secondly, the target tensor is divided into multiple blocks by the computing unit 102 in the artificial intelligence chip 100 and the aggregation features of each block are extracted; then the aggregation features of each block are hashed; that is, the computing unit 102 does not need to perform hash calculation on the entire byte of the target tensor, which not only effectively reduces the resource consumption of the computing unit, but also greatly improves the efficiency of hash calculation, thereby improving the efficiency of consistency verification.
[0066] In addition, hash calculations are performed on each block of the target tensor. This allows for further location of the biased blocks in the target tensor when inconsistent target tensors are found during consistency verification, thereby improving the accuracy of problem investigation and reducing the time spent on problem investigation.
[0067] In addition to the structure described above, the artificial intelligence chip 100 in this application may also include other structures, and this application does not specifically limit such structures.
[0068] Artificial intelligence chips 100 can be: Graphics Processing Unit (GPU), General-Purpose Graphics Processing Units (GPGPU), Domain Specific Architecture (DSA), etc.
[0069] The following is based on Figure 1 The diagram shown illustrates the architecture of an AI chip and details the process of a consistency verification method for model training. (See also...) Figure 2 This method is executed by an artificial intelligence chip and includes the following steps:
[0070] Step 201: Obtain multiple target tensors generated during model training.
[0071] Specifically, the consistency verification method for model training in this application can be applied to various scenarios, such as image processing, speech processing, and text processing. The physical meaning of the target tensor can differ in different application scenarios.
[0072] For example, in text processing scenarios, the target tensor can be text data used in tasks such as text generation and text recognition.
[0073] For example, in a speech processing scenario, the target tensor can be the speech data used in tasks such as speech enhancement, speech recognition, and speech synthesis.
[0074] For example, in image processing scenarios, the target tensor can be image data used in tasks such as image preprocessing, image segmentation, and object detection.
[0075] In this embodiment, the model training process includes multi-step iterative training, with each iterative training step corresponding to an iteration step identifier. Each iterative training step includes: forward computation, backward gradient computation, and parameter update. The target tensor generated during model training includes at least one of the following: a feature tensor generated by forward computation, a gradient tensor generated by backward gradient computation, and a weight tensor obtained by parameter update.
[0076] Step 202: For each target tensor, perform the verification operations shown in steps 2021-2024 respectively:
[0077] Step 2021: Divide the target tensor into multiple blocks and extract the aggregate features of each block.
[0078] Specifically, after dividing the target tensor into multiple blocks, a unique block identifier is assigned to each block. The block identifier can be a block number, a block name, etc.
[0079] In some embodiments, a baseline block size is obtained based on the memory bandwidth of the AI chip; then, the target tensor is divided into multiple blocks according to the baseline block size.
[0080] Specifically, the memory bandwidth of an AI chip is a hardware attribute of the AI chip, and the memory bandwidth may be different for different types of AI chips.
[0081] The baseline block size refers to the number of bytes occupied by a single block; for example, 32MB / block. When an AI chip processes data of the baseline block size, its processing performance can meet a preset performance state (e.g., optimal performance).
[0082] First, based on the base block size and the number of bytes occupied by each element, the number of elements in a single block is obtained. In this embodiment, the number of elements contained in a single block can be obtained using the following formula (1):
[0083] c_e = max(c_s / t_e_s, 4096) (1)
[0084] Where c_e represents the number of elements contained in a single block, c_s represents the base block size, t_e_s represents the number of bytes occupied by each element, and max represents the maximum value function.
[0085] In other words, first calculate the target ratio between the base block size and the number of bytes occupied by each element; when the target ratio is greater than 4096, the number of elements in a single block is the target ratio; when the target ratio is not greater than 4096, the number of elements in a single block is 4096, to ensure that each block contains at least 4096 elements.
[0086] Next, the target tensor is divided into multiple blocks based on the number of elements in a single block. Specifically, the total number of elements in the target tensor is divided by the number of elements in a single block, and the result is rounded up to obtain the number of blocks obtained from the division.
[0087] In this embodiment, based on the memory bandwidth of the artificial intelligence chip, a baseline block size is obtained, and the target tensor is divided into multiple blocks according to the baseline block size. In this way, when the artificial intelligence chip processes the blocks, it can make full use of hardware resources and improve the model training effect.
[0088] In some embodiments, the aggregation feature includes at least one of statistical features, location features, and seed features; wherein the statistical feature is obtained in the following manner:
[0089] For each block, determine the summation result and the mean of the multiple elements contained in the block, and obtain the statistical characteristics of the block based on the summation result and the mean.
[0090] Specifically, the methods for obtaining statistical features differ depending on the type of data being partitioned. The following explanation uses floating-point and integer partitioning as examples:
[0091] For floating-point blocks, the data type of the blocks is first converted to a specified floating-point type, such as any one of 32-bit single-precision floating-point (float32), 16-bit half-precision floating-point (float16), or 16-bit double-precision floating-point (bfloat16). Then, the summation of the multiple elements in the block is calculated using a summation operator, and the mean of the multiple elements is calculated using a mean operator. The summation result is amplified and rounded, and then the modulo of the rounded result is taken, retaining the lower 32 bits; the mean is amplified and rounded, and then the modulo of the rounded result is taken, retaining the lower 32 bits.
[0092] For integer blocks, first convert the data type of the block to a 32-bit signed integer (int32) type, then calculate the sum of the multiple elements contained in the block using the summation operator, and calculate the mean of the multiple elements using the mean operator; then take the modulo of the summation result and retain the lower 32 bits; and take the modulo of the mean and retain the lower 32 bits.
[0093] In this embodiment, natively supported data types (such as float32, float16, and int32) are used throughout the process to avoid dependence on non-native data types. Furthermore, after calculating statistical features, the numerical range is limited by a mask (such as retaining the lower 32 bits when taking the modulus), thereby improving compatibility.
[0094] In this embodiment of the application, the position features are obtained in the following way: the position features of the block are obtained based on the position index of the largest element and the position index of the smallest element in the block.
[0095] Specifically, the positional features of the blocks, also known as the extreme positional encoding of the blocks, are used to ensure the structural distinguishability between different target tensors.
[0096] For each block, obtain the position index of the maximum element in the block using the maximum value position (argmax) function or the top k extreme values (topk) function, and convert the position index to a Python integer; obtain the position index of the minimum element in the block using the argmin function or the topk function, and convert the position index to a Python integer.
[0097] The position index of the largest element is merged with the position index of the smallest element to obtain the position feature of the block; for example, the position index of the largest element is shifted left by 16 bits, occupying the high 16 bits of the 32-bit integer; the position index of the smallest element is shifted right by 16 bits, occupying the low 16 bits of the 32-bit integer; then the position index of the largest element shifted left by 16 bits is bitwise ORed with the position index of the smallest element shifted right by 16 bits to obtain the 32-bit position feature.
[0098] In addition, for the summation operator, the mean operator, and other operators used in the embodiments of this application, explicit synchronization of the operators is performed after execution to avoid dirty data reading caused by asynchronous scheduling.
[0099] In this embodiment of the application, the seed features are obtained in the following manner: the iterative training for generating the target tensor is determined from the multi-step iterative training included in the model training process, and the iteration step number identifier of the iterative training is obtained; then, based on the block identifier and the iteration step number identifier, the seed features of the block are obtained.
[0100] Specifically, the seed feature, also known as the scattering anti-collision feature, is used to characterize the uniqueness of the block and avoid the cancellation of block features. In practical applications, the iteration step number identifier associated with the target tensor is first obtained, and the iteration step number identifier is used as the basic seed; then, the block identifier i is added to the basic seed to obtain the seed feature of the block, where i is greater than 0, as shown in the following formula (2):
[0101] c_se = b_s + i a (2)
[0102] Where c_se represents the seed feature of the block, b_s represents the base seed, and a represents the golden ratio constant. Finally, the seed feature of the block is modulo-divided, retaining the lower 32 bits.
[0103] In this application, the target tensor is divided into blocks according to the granularity of hardware bandwidth adaptation. Then, numerical aggregation features (summation, mean, extreme value position, etc.) are extracted for each block. The numerical aggregation features are used to replace the full-byte hash calculation, which greatly reduces the amount of computation (specifically, it can be reduced to 1‰ of the original) while ensuring the accuracy of consistency verification.
[0104] Step 2022: Perform hash calculation on the aggregation features of each block to obtain the corresponding block hash value.
[0105] In the implementation of this application, the hash algorithms used for hash calculation include, but are not limited to: Message-Digest Algorithm 5 (MD5), Secure Hash Algorithm 2 (SHA-2), and Secure Hash Algorithm 3 (SHA-3).
[0106] In some embodiments, for each block, the statistical features, positional features and seed features of the block are weighted and fused to obtain the target fused features; then the target fused features are hashed to obtain the block hash value of the block.
[0107] Specifically, pre-set the magnitude weights for statistical features, location features, and seed features, with each magnitude weight corresponding to at least one amplification factor; the larger the magnitude weight, the larger the amplification factor. The statistical features, location features, and seed features are amplified according to the preset amplification factors, and then the amplified statistical features, amplified location features, and amplified seed features are fused to obtain the target fused feature.
[0108] For example, the magnitude weight of statistical features is set to 80%, the magnitude weight of location features to 10%, and the magnitude weight of seed features to 10%. In other words, statistical features play a dominant role, while location features and seed features play a supporting role.
[0109] The statistical features include: the summation result and the mean. The magnification factor corresponding to the summation result is set to 1000000000, so that the summation result occupies the high 32 bits; the magnification factor corresponding to the mean is set to 1000000, so that the mean occupies the middle 16 bits.
[0110] Set the magnification factor of the position feature to 10000, so that the position feature occupies the high 8 bits of the low 16 bits; set the magnification factor of the seed feature to 1, so that the seed feature occupies the low 8 bits of the low 16 bits.
[0111] In this way, the amplified statistical features, amplified positional features, and amplified seed features are merged into a 64-bit integer feature, which is the target fusion feature.
[0112] In some embodiments, to avoid the values in the target fusion features being too concentrated, this application uses the golden ratio to disperse the target fusion features and obtain updated target fusion features; specifically, the target fusion features can be multiplied by the golden ratio, and then the resulting multiplication can be truncated by taking the modulus to obtain updated target fusion features, so as to ensure that the values are evenly distributed.
[0113] Step 2023: Based on the obtained multiple block hash values, obtain the target hash value of the target tensor.
[0114] In some embodiments, to avoid numerical overflow and ensure uniform distribution of hash values, this application performs XOR aggregation on multiple block hash values to obtain the target hash value of the target tensor.
[0115] Specifically, the process involves sorting multiple block hash values to obtain a sorting result; selecting the first block hash value in the sorted result as the starting hash value; then, iterating through the remaining block hash values one by one according to the sorting result. For each block hash value encountered, an XOR operation is performed between that block hash value and the starting hash value to obtain an updated starting hash value. After traversing all block hash values, the final updated starting hash value is used as the target hash value of the target tensor.
[0116] In this embodiment of the application, multiple block hash values are XORed and aggregated to obtain the target hash value of the target tensor, which can eliminate the problem of numerical concentration in linear superposition and ensure that the hash values are evenly distributed.
[0117] It should be noted that, in the embodiments of this application, multiple block hash values can also be aggregated by summation to obtain the target hash value of the target tensor. This application does not specifically limit this method.
[0118] In some embodiments, different rounds of iterative training may generate the same target tensor, which may lead to duplicate hash values. Based on this, this application first performs XOR aggregation on multiple block hash values to obtain a first hash value; then, from the multi-step iterative training included in the model training process, it determines the iterative training that generates the target tensor and obtains the iteration step identifier of the iterative training; in addition, it obtains the hardware identifier of the hardware unit that generates the target tensor; wherein, the hardware unit that generates the target tensor may be: an artificial intelligence chip, a computing unit within the artificial intelligence chip, etc.; the hardware identifier of the hardware unit may be the number of the artificial intelligence chip, the number of the computing unit, etc.
[0119] The target tensor's shape information, iteration step number identifier, hardware identifier, and first hash value are fused together to obtain the target hash value of the target tensor. Among them, the shape information, iteration step number identifier, and hardware identifier serve as the target tensor's metadata and can uniquely identify the target tensor. In this way, incorporating the target tensor's metadata into the first hash value can effectively avoid hash value duplication and further enhance consistency across rounds.
[0120] In some embodiments, the shape information of the target tensor is hashed to obtain a second hash value; the iteration step identifier is hashed to obtain a third hash value; the hardware identifier is hashed to obtain a fourth hash value; and the first hash value, second hash value, third hash value and fourth hash value are XORed and aggregated to obtain the target hash value of the target tensor.
[0121] Specifically, the shape information of the target tensor is hashed and truncated modulo-based to obtain a 32-bit second hash value; the iteration step identifier is hashed and truncated modulo-based to obtain a 32-bit third hash value; the hardware identifier is hashed and truncated modulo-based to obtain a 32-bit fourth hash value. The second hash value is left-shifted by 32 bits, occupying the high 32 bits of the 64-bit integer. The 64-bit first hash value, the 32-bit second hash value (occupying the high 32 bits), the 32-bit third hash value (occupying the low 32 bits), and the 32-bit fourth hash value (occupying the low 32 bits) are XORed together to obtain the 64-bit target hash value.
[0122] In this embodiment, the target hash value is obtained by aggregating multi-dimensional features (statistical features, location features, seed features, and meta-information), which greatly reduces the probability of hash collisions and ensures hash accuracy, thereby meeting the consistency verification requirements for large model training.
[0123] Step 2024: Send the target hash value of the target tensor to the central processing unit so that the central processing unit can compare each received target hash value with the corresponding reference hash value to obtain the consistency verification result.
[0124] Specifically, the target hash value can be converted into an unsigned byte stream, and then the unsigned byte stream can be sent to the central processing unit for storage. For example, a 64-bit target hash value can be converted into an 8-byte unsigned byte stream, and then the 8-byte unsigned byte stream can be sent to the central processing unit.
[0125] In this embodiment, after obtaining multiple target hash values for the first model training, each target hash value is sent to the central processing unit (CPU). The CPU can use the received target hash values as reference hash values for subsequent consistency verification; of course, it can also update each reference hash value with the target hash values generated by other model training iterations, and this application does not specifically limit this.
[0126] In practical applications, a unique tensor identifier can be assigned to each target tensor generated during model training. This tensor identifier can be bound to the target hash value of the target tensor. For example, a consistency verification order for multiple target tensors can be pre-set, and the target tensors can be numbered according to the consistency verification order to obtain a unique tensor identifier for each target tensor.
[0127] For multiple model training sessions, the same tensor identifier allocation method can be set. In this way, when performing consistency verification on the target tensor generated by multiple model training sessions, multiple hash values bound to the same tensor identifier can be compared.
[0128] In other words, each reference hash value stored by the central processing unit (CPU) is bound to a tensor identifier. When the CPU receives the target hash values of multiple target tensors, it compares the reference hash values and target hash values corresponding to the same tensor identifier. If all target hash values are consistent, the consistency verification result is successful; if at least one target hash value is inconsistent, the consistency verification result is unsuccessful.
[0129] In some embodiments, when the consistency verification result is that the verification fails, in order to further investigate abnormal tensors, this application first obtains and compares the inconsistent target hash value and reference hash value; then, it re-hashes the target tensor corresponding to the target hash value to obtain the block hash values of multiple blocks; at the same time, it obtains the multiple block hash values associated with the reference hash value; then it compares the recalculated multiple block hash values with the multiple block hash values associated with the reference hash value one by one; if there are inconsistent block hash values, the block corresponding to the block hash value is regarded as an abnormal block.
[0130] In this embodiment, the hash calculation operation is first transferred from the CPU to the artificial intelligence chip. In this way, the artificial intelligence chip only needs to copy the target hash value of the target tensor to the CPU, instead of copying the entire target tensor. This greatly reduces the amount of data copied, thereby reducing the resource consumption of the artificial intelligence chip and improving the model training throughput.
[0131] Secondly, the target tensor is divided into multiple blocks, and the aggregate features of each block are extracted. Then, a hash calculation is performed on the aggregate features of each block, instead of hashing the entire bytes of the target tensor. This greatly improves the efficiency of hash calculation, thereby improving the efficiency of consistency verification.
[0132] In addition, hash calculations are performed on each block of the target tensor. This allows for the further location of the biased blocks in the target tensor when inconsistent target tensors are found during consistency verification. This transforms the investigation of inconsistencies from "full recalculation" to "block location". As a result, the investigation time can be reduced from hours to minutes, which not only greatly reduces the time spent on problem investigation but also improves the accuracy of problem investigation.
[0133] In some embodiments, the model training process and the validation operation are performed asynchronously.
[0134] Specifically, for each iteration of training in the model training process, a hash task is triggered every time at least one target tensor is generated. The hash task is then submitted asynchronously through a thread pool. At the same time, the future object of the returned hash task is received and stored in a dictionary, with "step + name" as the key identifier.
[0135] In this embodiment, the future object is a placeholder for the hash task (i.e., an asynchronous task), and returns immediately after the hash task is submitted without waiting for the hash task to complete; that is, the training process does not need to wait for the hash result, so the hash task will not block the model training process.
[0136] After a hashing task is submitted, a dedicated hashing pipeline performs hash-related operations (i.e., the process of obtaining the target hash value described above). This dedicated hashing pipeline is isolated from the training pipeline. Furthermore, after obtaining each target hash value through the dedicated hashing pipeline, each target hash value is sent to the central processing unit (CPU). The CPU's background thread pool asynchronously receives and saves each target hash value. Simultaneously, the received target hash values are compared with the corresponding reference hash values to obtain the comparison results.
[0137] Artificial intelligence chips can obtain the processing results of each hash task from the CPU through future objects during training intervals or after training (i.e., the comparison results between the target hash value and the corresponding reference hash value).
[0138] For example, see Figure 3 The GPU includes isolated computation streams and hash streams. Each iteration of the model training process is performed through the computation stream, and each iteration includes: forward computation, backward gradient computation, and parameter update.
[0139] The hash stream performs hash-related operations on each target tensor generated during model training. These operations include block partitioning, feature extraction, and hash calculation. After calculating the target hash value of the target tensor, the hash stream sends the target hash value to the GPU's background thread pool.
[0140] The GPU's background thread pool asynchronously receives and saves each target hash value; then it compares the received target hash value with the corresponding saved reference hash value to obtain the consistency verification result.
[0141] In this embodiment, firstly, the GPU performs hash-related operations on the target tensor without copying the entire target tensor to the CPU. This reduces the time taken for tensor hash calculation from seconds to less than 10ms, while also significantly reducing training throughput loss.
[0142] Secondly, the hash calculation and training process are executed in parallel, and the training process does not need to wait for the hash result, which greatly reduces the risk of training blockage. At the same time, this application adopts a minimally invasive design, which only requires adding asynchronous hash submission logic to the training framework without modifying the core training code, thus reducing deployment costs.
[0143] Based on the same technical concept, this application provides a schematic diagram of a consistency verification device for model training, applied to artificial intelligence chips, such as... Figure 4 As shown, the consistency verification device 400 for model training includes:
[0144] Module 401 is used to acquire multiple target tensors generated during model training.
[0145] Verification module 402 is used to perform the following verification operations for each target tensor:
[0146] The target tensor is divided into multiple blocks, and the aggregate features of each block are extracted;
[0147] The aggregation features of each block are hashed to obtain the corresponding block hash value;
[0148] Based on the obtained multiple block hash values, the target hash value of the target tensor is obtained;
[0149] The target hash value of the target tensor is sent to the central processing unit, so that the central processing unit compares each received target hash value with the corresponding reference hash value to obtain a consistency verification result.
[0150] Optionally, the verification module 402 is specifically used for:
[0151] The baseline block size is obtained based on the memory bandwidth of the AI chip.
[0152] The target tensor is divided into multiple blocks according to the aforementioned base block size.
[0153] Optionally, the aggregation feature includes at least one of statistical features, location features, and seed features;
[0154] The verification module 402 is specifically used for:
[0155] For each block, the summation result of multiple elements contained in the block and the mean of the multiple elements are determined, and the statistical characteristics of the block are obtained based on the summation result and the mean.
[0156] Based on the position index of the largest element and the position index of the smallest element in the block, the positional features of the block are obtained;
[0157] From the multi-step iterative training included in the model training process, determine the iterative training that generates the target tensor, and obtain the iteration step identifier of the iterative training;
[0158] Based on the block identifier and the iteration step identifier, the seed features of the block are obtained.
[0159] Optionally, the verification module 402 is specifically used for:
[0160] For each block, the statistical features, positional features, and seed features of the block are weighted and fused to obtain the target fusion features;
[0161] The target fusion feature is hashed to obtain the block hash value of the block.
[0162] Optionally, the verification module 402 is specifically used for:
[0163] The target hash value of the target tensor is obtained by XOR aggregation of the multiple block hash values.
[0164] Optionally, the verification module is specifically used for:
[0165] The multiple block hash values are XORed and aggregated to obtain the first hash value;
[0166] From the multi-step iterative training included in the model training process, determine the iterative training that generates the target tensor, and obtain the iteration step identifier of the iterative training;
[0167] Obtain the hardware identifier of the hardware unit that generates the target tensor;
[0168] The shape information of the target tensor, the iteration step identifier, the hardware identifier, and the first hash value are fused together to obtain the target hash value of the target tensor.
[0169] Optionally, the verification module 402 is specifically used for:
[0170] The shape information of the target tensor is hashed to obtain a second hash value;
[0171] Perform a hash calculation on the iteration step identifier to obtain a third hash value;
[0172] Perform a hash calculation on the hardware identifier to obtain a fourth hash value;
[0173] The first hash value, the second hash value, the third hash value, and the fourth hash value are XORed together to obtain the target hash value of the target tensor.
[0174] Optionally, the model training process and the verification operation are executed asynchronously.
[0175] In this embodiment, the hash calculation operation is first transferred from the CPU to the artificial intelligence chip. In this way, the artificial intelligence chip only needs to copy the target hash value of the target tensor to the CPU, instead of copying the entire target tensor. This greatly reduces the amount of data copied, thereby reducing the resource consumption of the artificial intelligence chip and improving the model training throughput.
[0176] Secondly, the target tensor is divided into multiple blocks, and the aggregate features of each block are extracted. Then, a hash calculation is performed on the aggregate features of each block, instead of hashing the entire bytes of the target tensor. This greatly improves the efficiency of hash calculation, thereby improving the efficiency of consistency verification.
[0177] In addition, hash calculations are performed on each block of the target tensor. This allows for the further location of the biased blocks in the target tensor when inconsistent target tensors are found during consistency verification. This transforms the investigation of inconsistencies from "full recalculation" to "block location". As a result, the investigation time can be reduced from hours to minutes, which not only greatly reduces the time spent on problem investigation but also improves the accuracy of problem investigation.
[0178] In the embodiments of this application, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.
[0179] Based on the same technical concept, embodiments of this application provide a computer device, such as... Figure 5 As shown, it includes at least one artificial intelligence chip 100 and a memory 501 connected to at least one artificial intelligence chip 100. In this embodiment, the specific connection medium between the artificial intelligence chip 100 and the memory 501 is not limited. Figure 5 Taking the connection between the AI chip 100 and the memory 501 via a bus as an example, the bus can be divided into address bus, data bus, control bus, etc.
[0180] In this embodiment of the application, the memory 501 stores instructions that can be executed by at least one artificial intelligence chip 100. By executing the instructions stored in the memory 501, at least one artificial intelligence chip 100 can perform the steps of the above-described model training consistency verification method.
[0181] The artificial intelligence chip 100 serves as the control center of the computer device. It connects to various parts of the computer device via various interfaces and lines, and performs consistency verification of model training by running or executing instructions stored in the memory 501 and accessing data stored in the memory 501. Optionally, the artificial intelligence chip 100 may include one or more processing units. The artificial intelligence chip 100 may integrate an application processor and a modem processor. The application processor primarily handles the operating system, user interface, and applications, while the modem processor primarily handles wireless communication. It is understood that the modem processor may not be integrated into the artificial intelligence chip 100. In some embodiments, the artificial intelligence chip 100 and the memory 501 may be implemented on the same chip; in other embodiments, they may be implemented on separate chips.
[0182] The artificial intelligence chip 100 can be a general-purpose processor, such as a central processing unit (CPU), digital signal processor, application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), or other programmable logic device, discrete gate or transistor logic device, or discrete hardware component, capable of implementing or executing the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly manifested as being executed by a hardware processor, or executed by a combination of hardware and software modules within the processor.
[0183] Memory 501, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules. Memory 501 may include at least one type of storage medium, such as flash memory, hard disk, multimedia card, card-type memory, random access memory (RAM), static random access memory (SRAM), programmable read-only memory (PROM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), magnetic storage, magnetic disk, optical disk, etc. Memory 501 can be any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer device, but is not limited thereto. Memory 501 in the embodiments of this application may also be a circuit or any other device capable of implementing storage functions for storing program instructions and / or data.
[0184] Based on the same inventive concept, embodiments of this application provide a computer-readable storage medium storing a computer program executable by a computer device, which, when run on the computer device, causes the computer device to perform the steps of the consistency verification method for model training described above.
[0185] Based on the same inventive concept, this application provides a computer program product, which includes a computer program stored on a computer-readable storage medium. The computer program includes program instructions that, when executed by a computer device, cause the computer device to perform the steps of the above-described model training consistency verification method.
[0186] Those skilled in the art will understand that embodiments of the present invention can be provided as methods or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0187] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer apparatus or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.
[0188] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer device or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.
[0189] These computer program instructions may also be loaded onto a computer device or other programmable data processing equipment to cause a series of operational steps to be performed on the computer device or other programmable equipment to produce a process implemented by the computer device, thereby providing instructions that execute on the computer device or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.
[0190] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the invention.
[0191] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.
Claims
1. A method for verifying the consistency of model training, characterized in that, Applied to artificial intelligence chips, the method includes: Obtain multiple target tensors generated during model training; For each target tensor, perform the following verification operations: The target tensor is divided into multiple blocks, and the aggregate features of each block are extracted. The aggregate features include seed features. The seed features are obtained by: determining the iterative training that generates the target tensor from the multi-step iterative training included in the model training process, and obtaining the iteration step number identifier of the iterative training; and obtaining the seed features of the block based on the block identifier and the iteration step number identifier. The aggregation features of each block are hashed to obtain the corresponding block hash value; Based on the obtained multiple block hash values, the target hash value of the target tensor is obtained; The target hash value of the target tensor is sent to the central processing unit (CPU) so that the CPU compares each received target hash value with the corresponding reference hash value to obtain a consistency verification result; the reference hash value is the target hash value generated by the CPU during the first model training or other model training sessions.
2. The method as described in claim 1, characterized in that, The step of dividing the target tensor into multiple blocks includes: The baseline block size is obtained based on the memory bandwidth of the AI chip. The target tensor is divided into multiple blocks according to the aforementioned base block size.
3. The method as described in claim 1, characterized in that, The aggregation feature also includes at least one of statistical features and location features; The statistical features and the location features are obtained in the following manner: For each block, the summation result of multiple elements contained in the block and the mean of the multiple elements are determined, and the statistical characteristics of the block are obtained based on the summation result and the mean. The positional features of the block are obtained based on the position indices of the largest and smallest elements in the block.
4. The method as described in claim 3, characterized in that, The step of performing hash calculations on the aggregation features of each block to obtain the corresponding block hash value includes: For each block, the statistical features, positional features, and seed features of the block are weighted and fused to obtain the target fusion features; The target fusion feature is hashed to obtain the block hash value of the block.
5. The method as described in claim 1, characterized in that, The step of obtaining the target hash value of the target tensor based on the obtained multiple block hash values includes: The target hash value of the target tensor is obtained by XOR aggregation of the multiple block hash values.
6. The method as described in claim 5, characterized in that, The step of XORing and aggregating the multiple block hash values to obtain the target hash value of the target tensor includes: The multiple block hash values are XORed and aggregated to obtain the first hash value; From the multi-step iterative training included in the model training process, determine the iterative training that generates the target tensor, and obtain the iteration step identifier of the iterative training; Obtain the hardware identifier of the hardware unit that generates the target tensor; The shape information of the target tensor, the iteration step identifier, the hardware identifier, and the first hash value are fused together to obtain the target hash value of the target tensor.
7. The method as described in claim 6, characterized in that, The step of fusing the shape information of the target tensor, the iteration step identifier, the hardware identifier, and the first hash value to obtain the target hash value of the target tensor includes: The shape information of the target tensor is hashed to obtain a second hash value; Perform a hash calculation on the iteration step identifier to obtain a third hash value; Perform a hash calculation on the hardware identifier to obtain a fourth hash value; The first hash value, the second hash value, the third hash value, and the fourth hash value are XORed together to obtain the target hash value of the target tensor.
8. The method according to any one of claims 1 to 7, characterized in that, The model training process and the verification operation are executed asynchronously.
9. A computer device comprising a memory, an artificial intelligence chip, and a computer program stored in the memory and running on the artificial intelligence chip, characterized in that, When the artificial intelligence chip executes the computer program, it implements the steps of the method as described in any one of claims 1 to 7.
10. A computer-readable storage medium, characterized in that, It stores a computer program that is executed by a computer device, which, when run on the computer device, causes the computer device to perform the steps of the method as described in any one of claims 1 to 7.