Precision calibration methods, devices, electronic equipment, and media between heterogeneous chips

Abstract

This disclosure relates to a method, apparatus, electronic device, and medium for precision calibration among heterogeneous chips, applied to a precision calibration system comprising multiple heterogeneous chips. The method includes: sending a precision test command to each of the heterogeneous chips, the precision test command instructing each heterogeneous chip to run a target model; receiving the computation results for the target model sent by each heterogeneous chip, comparing the computation results with standard computation results to obtain a precision detection report, the precision detection report including precision error information of each heterogeneous chip; and calibrating the computational precision of the multiple heterogeneous chips based on the precision detection report. This disclosure can accurately adjust the computational precision of each heterogeneous chip by utilizing the generated precision detection report, thus ensuring the accuracy of model training.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of artificial intelligence, in particular, to a precision calibration method and device between heterogeneous chips, an electronic device and a medium. BACKGROUND

[0002] With the development of artificial intelligence (AI), the scale and complexity of AI models are increasing, and the computing resource requirements for AI model training are also growing. In order to improve the efficiency and speed of AI model training, distributed training using heterogeneous systems has gradually become a trend.

[0003] However, different hardware architectures have different computing capabilities and precision characteristics, which leads to inconsistent performance and precision across hardware platforms, thereby affecting the reliability of the model training results. SUMMARY

[0004] To overcome the problems in the related art, the present disclosure provides a precision calibration method and device between heterogeneous chips, an electronic device and a medium

[0005] According to a first aspect of an embodiment of the present disclosure, a precision calibration method between heterogeneous chips is provided, applied to a precision calibration system, the precision calibration system comprising a plurality of heterogeneous chips, and the method comprising:

[0006] sending a precision test instruction to each of the heterogeneous chips, the precision test instruction being used to instruct each of the heterogeneous chips to run a target model;

[0007] receiving an operation result of the target model sent by each of the heterogeneous chips, and comparing the operation result with a standard operation result to obtain a precision detection report, the precision detection report comprising precision error information of each of the heterogeneous chips;

[0008] calibrating the computing precision of the plurality of heterogeneous chips based on the precision detection report.

[0009] Optionally, after calibrating the computing precision of the plurality of heterogeneous chips based on the precision detection report, the method comprises:

[0010] instructing the plurality of heterogeneous chips to cooperatively train the target model;

[0011] for training of the target model, collecting intermediate calculation results of each of the heterogeneous chips;

[0012] comparing the intermediate calculation results of the plurality of heterogeneous chips to obtain a comparison result;

[0013] According to the comparison result, consistency of the plurality of heterogeneous chips is verified.

[0014] Optionally, the consistency verification of the plurality of heterogeneous chips according to the comparison result comprises:

[0015] It is determined whether there is a first chip in the plurality of heterogeneous chips whose comparison result exceeds a preset error.

[0016] If it is determined that the first chip exists, the training of the target model is terminated, and the calculation accuracy of the first chip is recalibrated.

[0017] Optionally, if it is determined that the first chip exists, the training of the target model is terminated, and the calculation accuracy of the first chip is recalibrated, comprising:

[0018] When it is determined that the first chip exists in the plurality of heterogeneous chips, if it is determined that the first chip performs a specified task, the accuracy correction strength of the first chip is increased.

[0019] Optionally, when it is determined that the first chip exists in the plurality of heterogeneous chips, if it is determined that the first chip performs a specified task, the allocation mode of the specified task is adjusted, and the second chip is instructed to perform the specified task, the calculation accuracy of the second chip is higher than that of the first chip.

[0020] Optionally, before the precision test instruction is sent to each of the plurality of heterogeneous chips, comprising:

[0021] A computation graph analysis is performed on the target model to obtain a computation graph analysis result.

[0022] According to the computation graph analysis result, node attribute analysis and key operator identification are performed to obtain a critical path analysis result.

[0023] According to the critical path analysis result, a burying point strategy is generated, and a burying point is inserted in the target model based on the burying point strategy.

[0024] Optionally, the operation result of each of the plurality of heterogeneous chips for the target model is received, and the operation result is compared with a standard operation result to obtain a precision detection report, comprising:

[0025] The operation result of each of the plurality of heterogeneous chips for the target model at the burying point is received.

[0026] The error between the operation result and the standard operation result is obtained, and the difference error is written into the precision detection report.

[0027] Optionally, writing the error into the accuracy detection report includes:

[0028] When it is determined that the error exceeds the preset error, the error is marked and the marked error is written into the accuracy detection report.

[0029] Optionally, the method further includes:

[0030] When it is determined that the error exceeds the preset error, dynamic precision compensation is performed on the calculation result.

[0031] According to a second aspect of the present disclosure, a precision calibration apparatus for heterogeneous chips is provided, applied to a precision calibration system, the precision calibration system including multiple heterogeneous chips, the apparatus comprising:

[0032] The sending module is configured to send a precision test command to each of the heterogeneous chips, the precision test command being used to instruct each of the heterogeneous chips to run the target model;

[0033] The generation module is configured to receive the calculation results for the target model sent by each of the heterogeneous chips, compare the calculation results with the standard calculation results, and obtain an accuracy detection report, wherein the accuracy detection report includes the accuracy error information of each of the heterogeneous chips;

[0034] The calibration module is configured to calibrate the computational accuracy of the plurality of heterogeneous chips based on the accuracy detection report.

[0035] According to a third aspect of the present disclosure, a computer-readable storage medium is provided, having stored thereon computer program instructions that, when executed by a processor, implement the steps of the precision calibration method between heterogeneous chips provided in the first aspect of the present disclosure.

[0036] According to a fourth aspect of the present disclosure, an electronic device is provided, comprising:

[0037] A memory on which computer programs are stored;

[0038] A processor is configured to execute the computer program in the memory to implement the steps of the precision calibration method between heterogeneous chips provided in the first aspect of this disclosure.

[0039] According to a fifth aspect of the present disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps of the precision calibration method between heterogeneous chips provided in the first aspect of the present disclosure.

[0040] This disclosure improves the accuracy of target model training by calibrating the computational precision of multiple heterogeneous chips. Specifically, a precision test command is sent to each heterogeneous chip to instruct it to run the target model. Based on this, the computational results for the target model sent by each heterogeneous chip are received and compared with standard computation results to obtain a precision detection report. The precision detection report includes the precision error information of each heterogeneous chip. Finally, based on this precision detection report, the computational precision of multiple heterogeneous chips can be accurately calibrated. Since the precision detection report is obtained by running the target model on each heterogeneous chip, chip precision alignment can be achieved more accurately and efficiently based on this report.

[0041] Other features and advantages of this disclosure will be described in detail in the following detailed description section. Attached Figure Description

[0042] The accompanying drawings are provided to further illustrate the present disclosure and form part of the specification. They are used together with the following detailed description to explain the present disclosure, but do not constitute a limitation thereof. In the drawings:

[0043] Figure 1 This is a flowchart illustrating a precision calibration method between heterogeneous chips according to an exemplary embodiment.

[0044] Figure 2 This is a flowchart illustrating another method for precision calibration between heterogeneous chips according to an exemplary embodiment.

[0045] Figure 3 This is a flowchart illustrating an exemplary method for generating a precision detection report in another precision calibration method between heterogeneous chips, according to an exemplary embodiment.

[0046] Figure 4 This is a structural example diagram of the target model in another precision calibration method between heterogeneous chips, according to an exemplary embodiment.

[0047] Figure 5 This is an example diagram of the computational graph corresponding to the target model in another precision calibration method between heterogeneous chips, according to an exemplary embodiment.

[0048] Figure 6 This is an example diagram of an anomaly in another precision calibration method between heterogeneous chips, according to an exemplary embodiment.

[0049] Figure 7 This is a block diagram illustrating a precision calibration device between heterogeneous chips according to an exemplary embodiment.

[0050] Figure 8This is a block diagram illustrating an electronic device according to an exemplary embodiment.

[0051] Figure 9 This is a block diagram illustrating a server according to an exemplary embodiment. Detailed Implementation

[0052] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0053] In the description of this disclosure, terms such as "first" and "second" are used to distinguish similar objects and should not be construed as indicating a specific order or sequence. Furthermore, unless otherwise stated, in the description with reference to the accompanying drawings, the same reference numerals in different drawings denote the same elements.

[0054] Although operations or steps are described in a specific order in the accompanying drawings in the embodiments of this disclosure, it should not be construed as requiring these operations or steps to be performed in the specific order or serial order shown, or requiring all of the shown operations or steps to be performed to obtain the desired result. In the embodiments of this disclosure, these operations or steps may be performed serially; they may be performed in parallel; or a portion of these operations or steps may be performed.

[0055] As described above, in order to improve training efficiency and speed, more and more computing architectures are adopting heterogeneous systems for distributed training. These heterogeneous systems can include CPUs (Central Processing Units), GPUs (Graphics Processing Units), FPGAs (Field-Programmable Gate Arrays), and dedicated AI accelerators.

[0056] However, different hardware architectures possess varying computational capabilities and precision characteristics, particularly in floating-point and integer arithmetic precision, which can exhibit significant differences. These differences can easily lead to inconsistencies in model performance and accuracy during cross-platform training tasks, thereby affecting the reliability of training results.

[0057] In related technologies, while heterogeneous computing architectures can optimize the utilization efficiency of different computing resources through task allocation and data parallelism, ensuring the consistency of computing results across multiple hardware platforms remains a pressing issue. For example, some hardware may introduce errors due to precision limitations when performing floating-point calculations. If these errors are not corrected in time, they may accumulate during training, leading to a decline in the performance of the final trained model.

[0058] In addition, the relevant technologies mainly improve the training efficiency of heterogeneous systems by optimizing time and energy consumption, and rarely involve how to effectively correct the accuracy error between hardware platforms. That is, there is currently no effective means to solve the problem of computational accuracy alignment between heterogeneous chips. In particular, when multiple hardware platforms are used to train AI models in collaboration, there is still a lot of room for improvement in how to effectively control and correct the accuracy error between different chips.

[0059] To address the aforementioned issues, this disclosure proposes a precision calibration method for heterogeneous chips. This method calibrates the computational precision of multiple heterogeneous chips using a pre-generated precision detection report, thereby effectively correcting precision errors between different chips and improving the accuracy of artificial intelligence model training.

[0060] Figure 1 This is a flowchart illustrating a precision calibration method between heterogeneous chips according to an exemplary embodiment, such as... Figure 1 As shown, the method may include the following steps.

[0061] In step S110, a precision test command is sent to each heterogeneous chip.

[0062] This disclosure can be applied to an accuracy calibration system, which may include multiple heterogeneous chips that can collaboratively train a target model. For example, the accuracy calibration system may include a first heterogeneous chip and a second heterogeneous chip, wherein the first heterogeneous chip is used to train layers 7-12 of the target model's computational network, and the second heterogeneous chip is used to train layers 1-6 of the target model's computational network.

[0063] In this embodiment, the precision calibration system may include not only multiple heterogeneous chips, but also a main control device, which may be an electronic device or a server. Here, the main control device can be used to perform precision calibration between heterogeneous chips; specifically, the main control device can send precision test commands to each heterogeneous chip.

[0064] As an alternative approach, before generating an accuracy test report or before training the target model, embodiments of this disclosure may send an accuracy test instruction to each heterogeneous chip to instruct each heterogeneous chip to run the target model.

[0065] The accuracy test command can be triggered manually by the user or by the accuracy calibration system when preset conditions are met. For example, the accuracy calibration system can trigger the sending of the accuracy test command when it receives a scheduled task. Alternatively, the accuracy calibration system can trigger the sending of the accuracy test command when it receives a model training command or detects a training data acquisition command. There are no explicit restrictions on how the accuracy test command is triggered; it can be selected based on the actual situation.

[0066] In some implementations, precision testing instructions are used to analyze the computational precision of each heterogeneous chip. Specifically, precision testing instructions can be used to analyze at least one computational precision of each heterogeneous chip, such as floating-point precision, integer precision, rounding error precision, and numerical overflow precision.

[0067] As an alternative approach, the accuracy test instructions may include a program for testing, which can be a program related to the target model. That is, the accuracy test instructions may include the target model. By sending the target model to each heterogeneous chip, each heterogeneous chip can be instructed to execute the target model to obtain the calculation results. Here, the execution of the target model may involve addition, multiplication, matrix multiplication, and dot product, etc.

[0068] Optionally, the accuracy test command may also include standard operator operations. By sending standard operator sub-operations to each heterogeneous chip, each heterogeneous chip can execute a series of standard operator operations. Here, standard operator operations may include addition, multiplication, matrix multiplication, etc.

[0069] Optionally, the accuracy test instruction may also include an address for obtaining the operation. That is, when the accuracy test instruction does not include a specified operation, it can be determined whether the instruction carries an address for storing the specified operation, and the specified operation can be found based on that address. On this basis, the heterogeneous chip can execute the specified operation to obtain the operation result.

[0070] After receiving the accuracy test command, each heterogeneous chip in the accuracy calibration system can execute the calculations in the accuracy test command, such as running the specified calculations indicated by the heterogeneous chip. These specified calculations can be target model calculations or standard operator calculations.

[0071] It should be noted that the computation received by each heterogeneous chip can be the same. That is, the embodiments of this disclosure can instruct multiple heterogeneous chips to perform the same computation at the same time and obtain multiple computation results, so as to obtain a more accurate accuracy detection report.

[0072] In step S120, the calculation results for the target model sent by each heterogeneous chip are received, and the calculation results are compared with the standard calculation results to obtain an accuracy detection report.

[0073] As an optional approach, after sending an accuracy test command to each heterogeneous chip, this embodiment of the disclosure can detect whether it has received the calculation results for the target model sent by each heterogeneous chip. Upon receiving the calculation results for the target model sent by each heterogeneous chip, this embodiment of the disclosure can compare the calculation results of each heterogeneous chip with the standard calculation results to obtain an accuracy test report.

[0074] In other words, the embodiments of this disclosure can compare the intermediate calculation results of multiple heterogeneous chips with the standard calculation results respectively, and obtain multiple comparison results, which can be accuracy error information.

[0075] Here, the standard computation result can be a pre-set computation result, which can be the result obtained by running the target model on a standard device. For example, embodiments of this disclosure can obtain correct data by running the same code on an NV / CPU, and this correct data can then be used as the standard computation result.

[0076] In this embodiment of the disclosure, the accuracy detection report may include accuracy error information for each heterogeneous chip, meaning that there may be a corresponding relationship between the accuracy error information and the heterogeneous chips. Furthermore, the same heterogeneous chip may correspond to multiple sub-error information, which may include sub-error information for addition operations, sub-error information for multiplication operations, or sub-error information for matrix multiplication operations.

[0077] To better illustrate the contents of the accuracy test report, this disclosure provides the information shown in Table 1 below:

[0078] Table 1

[0079]

[0080] The above calculation errors are for illustrative purposes only and are not intended as actual limitations. The actual situation shall prevail.

[0081] In this embodiment of the disclosure, the errors in addition, multiplication, and matrix multiplication are all obtained by comparing the calculation results of the heterogeneous chip with the standard calculation results. For example, when chip 1 calculates the sum of two floating-point numbers 1.00000001 and 0.99999999, the calculated result is 2.00000001, while the standard calculation result is 2.00000000. The error between the two is 2.00000001 − 2.00000000 = 0.00000001, that is, the addition error of chip 1 is 0.00000001.

[0082] The generated hardware accuracy report (accuracy test report) can clearly show the differences in computing accuracy between various chips, which can significantly improve the transparency of accuracy alignment and thus improve the efficiency of accuracy test.

[0083] In step S130, the calculation accuracy of multiple heterogeneous chips is calibrated based on the accuracy detection report.

[0084] As an optional approach, after generating the accuracy detection report, embodiments of this disclosure can calibrate the computational accuracy of multiple heterogeneous chips based on the accuracy detection report. Specifically, the actual computation results are compensated based on the accuracy error information corresponding to each heterogeneous chip. For example, when chip 1 is training the target model and performing addition operations, the real-time acquired computation results can be compensated based on the addition operation error in the accuracy detection report.

[0085] By testing the computational accuracy of different hardware architectures (heterogeneous chips), the embodiments of this disclosure can obtain the computational error of each hardware architecture and perform targeted calibration, thus ensuring the consistency of the collaborative training results of different hardware architectures.

[0086] In one specific embodiment, the computational precision calibration method may include, but is not limited to, increasing floating-point digits, adjusting rounding strategies, or applying high-precision arithmetic units. Through precision calibration, the computational results of each heterogeneous chip can be kept consistent with the set precision target.

[0087] This embodiment of the disclosure improves the accuracy of target model training by calibrating the computational precision of multiple heterogeneous chips. Specifically, a precision test command is sent to each heterogeneous chip to instruct it to run the target model. Based on this, the computational results for the target model sent by each heterogeneous chip are received and compared with standard computation results to obtain a precision detection report. The precision detection report includes the precision error information of each heterogeneous chip. Finally, based on the precision detection report, the computational precision of multiple heterogeneous chips can be accurately calibrated. Since the precision detection report is obtained by running the target model on each heterogeneous chip, chip precision alignment can be achieved more accurately and efficiently based on the report.

[0088] Figure 2 This is a flowchart illustrating another precision calibration method between heterogeneous chips according to an exemplary embodiment, such as... Figure 2 As shown, the method may include the following steps.

[0089] In step S210, a precision test command is sent to each heterogeneous chip.

[0090] As an optional approach, in the process of generating the accuracy test report, this embodiment of the disclosure can first perform an automatic data entry operation, collect the calculation results of each heterogeneous chip for the target model based on the inserted data entry points, and obtain a more accurate accuracy test report based on the calculation results.

[0091] Here, the specific generation of the accuracy test report can be as follows: Figure 3 As shown, based on Figure 3 As can be seen, after obtaining the target model, the embodiments of this disclosure can first perform static analysis on the computational graph of the target model. For example, the network structure of the target model is as follows: Figure 4 As shown, the target model at this time can include an input layer, a hidden layer, and an output layer.

[0092] Based on this, the embodiments disclosed herein can obtain Figure 4 The computational graph of the target model shown can be as follows: Figure 5 As shown. At this point, it is possible to... Figure 5 The computation graph shown is subjected to node attribute analysis and key operator identification to obtain identification results. Based on these results, this embodiment of the disclosure can generate a corresponding event tracking strategy. On this basis, event tracking points are automatically inserted into the target model according to the event tracking strategy, that is, the event tracking points are inserted into the code of the target model.

[0093] As can be seen, the embodiments of this disclosure can perform computational graph analysis on the target model to obtain computational graph analysis results. Based on these results, node attribute analysis and key operator identification can be performed to obtain critical path analysis results. On this basis, a tracking strategy can be generated from the critical path analysis results, and tracking points can be inserted into the target model based on this strategy. These tracking points can be used for subsequent collection of computational results.

[0094] It should be noted that, in the embodiments of this disclosure, during the process of sending accuracy test commands to each heterogeneous chip, the embedded points can be inserted into the target model, or the accuracy test commands can be sent to the target model first, and then each heterogeneous chip can be instructed to insert the embedded points into the target model. There is no explicit limitation on when to insert the embedded points into the target model; it can be selected according to the actual situation.

[0095] In summary, the embodiments of this disclosure can automatically identify key nodes and insert monitoring points by performing static analysis on the computation graph, thereby achieving automatic capture and monitoring of the accuracy and performance of model operators.

[0096] In step S220, the calculation results for the target model sent by each heterogeneous chip are received, and the calculation results are compared with the standard calculation results to obtain an accuracy detection report.

[0097] In some implementations, after obtaining the target model containing the embedded points, each heterogeneous chip can run the target model, and during the running of the target model, the calculation results can be collected based on the automatically inserted embedded points. Through automatic embedded point and capture insertion technology, the embodiments of this disclosure can reduce the workload of developers manually inserting monitoring points, thereby improving the automation and efficiency of operator accuracy detection and debugging.

[0098] At this point, the main control device can receive the calculation results collected at the embedding points for the target model from each heterogeneous chip. Based on this, it obtains the error between each calculation result and the standard calculation result and writes the error into the accuracy detection report. Since the target model can automatically have multiple embedding points, each heterogeneous chip can send multiple calculation results, resulting in multiple corresponding errors. In the process of writing multiple errors into the accuracy detection report, this embodiment can write the errors to the corresponding locations of the heterogeneous chips.

[0099] For example, the multiplication and addition errors of the first heterogeneous chip can be written to the location of the first heterogeneous chip identifier 00; the multiplication and addition errors of the second heterogeneous chip can be written to the location of the second heterogeneous chip identifier 01.

[0100] It should be noted that, in the process of writing the error between the calculation result and the standard calculation result into the accuracy detection report, the embodiments of this disclosure can observe whether there is an accuracy anomaly, that is, observe the accuracy anomaly point.

[0101] Specifically, it is determined whether the error between the calculation result and the standard calculation result is greater than a preset error. If the error is greater than the preset error, this embodiment of the disclosure can mark the error and write the marked error into the accuracy detection report. In other words, when an accuracy anomaly is observed, this embodiment of the disclosure can highlight the accuracy anomaly code.

[0102] At this point, this embodiment of the present disclosure can perform dynamic accuracy compensation. That is, when it is determined that the error between the calculation result and the standard calculation result exceeds a preset error, this embodiment of the present disclosure can perform dynamic accuracy compensation on the calculation result. Afterwards, this embodiment of the present disclosure can run the remaining model to completion and record accuracy anomalies to obtain a final accuracy detection report. Here, accuracy anomalies can be as follows: Figure 6 As shown, the points indicated by markers 601, 602, and 603 are all points with abnormal accuracy, while the other points are points with normal accuracy.

[0103] As an example, when running the 6th layer of the target model's network, if a data point is found, the calculation result at that data point can be collected. If it is determined that the error between this calculation result and the standard calculation result exceeds the preset error, the data point can be recorded, that is, the code corresponding to the data point can be marked as an accuracy anomaly.

[0104] Meanwhile, the embodiments of this disclosure can perform dynamic precision compensation on the calculation results to avoid error accumulation and affecting the acquisition of subsequent calculation results. When encountering precision errors, the reference precision (standard calculation results, such as NV or CPU precision) is automatically used to compensate for subsequent calculations, avoiding the accumulation of precision errors. The dynamic precision compensation mechanism can identify the precision error of the entire network in one operation, avoiding multiple network debugging operations, reducing the number of debugging operations from O(n) to O(1), and significantly shortening the debugging time.

[0105] It should be noted that the accuracy test report can be obtained before the target model is trained. That is, before training the target model, the embodiments of this disclosure can first analyze the computational accuracy of each heterogeneous chip. The specific analysis content includes floating-point operation accuracy, integer operation accuracy, rounding error, numerical overflow, etc.

[0106] By setting a standard test set, this embodiment of the disclosure can perform test calculations on each chip separately, collecting deviation data for each chip in floating-point and integer numerical operations. Based on this, a precision characteristic analysis report (precision detection report) is generated according to the analysis results. This report can be used to describe the error distribution and error range of each chip in different operations. In other words, by analyzing the basic computational operators, it is possible to obtain reports on the computational performance and precision of operators on different chips.

[0107] As a specific implementation method, before training the target model, embodiments of this disclosure can analyze and collect the accuracy characteristics of different heterogeneous chips. Specifically, standard operator test operations are performed, instructing the heterogeneous chips to perform a series of standard operator operations (running the target model), such as matrix multiplication, dot product, etc., and collecting the execution results of these operators (operation results).

[0108] Based on this, error analysis is performed, and the collected chip execution results are compared with standard precision results (standard calculation results) to obtain the error range of each chip in floating-point and integer operations, and obtain the error analysis results. The error analysis results can include error ranges such as calculation precision, rounding error, and overflow.

[0109] Next, a precision feature report (precision detection report) is generated based on the error analysis results. The precision detection report can include the precision error range and calculation deviation of each operator on different chips. This precision detection report can be used as a reference for subsequent precision alignment.

[0110] In summary, the embodiments of this disclosure can obtain an accuracy detection report by collecting the accuracy features of heterogeneous chips. The process of generating the accuracy detection report can be based on a specific test set or labeled data (target model), and the operator tasks performed by each heterogeneous chip can be the same, thus ensuring the accuracy of the accuracy features.

[0111] It should be noted that, in the process of generating the accuracy test report, the embodiments of this disclosure can consider the impact of chip operating frequency, temperature, running time, etc., on accuracy to ensure the long-term stability of the collected accuracy features. For example, the embodiments of this disclosure can instruct heterogeneous chips to run the target model under extreme conditions, such as running the target model when the temperature exceeds a preset temperature, thus avoiding the impact of extreme environments on accuracy calibration.

[0112] It should also be noted that, after obtaining the comparison result between the calculation result and the standard calculation result, the embodiments of this disclosure can classify the accuracy error of each chip. By comparing the standard calculation result with the calculation result output by the chip, the typical error of each operation (such as addition, multiplication, matrix multiplication, etc.) can be determined. Here, the classification can include normal error and abnormal error. In addition, for calculation operations with large accuracy errors, the embodiments of this disclosure can record their error range and generate an accuracy detection report for subsequent accuracy correction.

[0113] In step S230, the computational accuracy of multiple heterogeneous chips is calibrated based on the accuracy detection report.

[0114] As an optional approach, after obtaining the accuracy detection report, embodiments of this disclosure can calibrate the computational accuracy of each heterogeneous chip based on the accuracy detection report. Here, the calibration of computational accuracy can be performed before the target model is trained, or it can be adjusted in real time during the target model training process. The calibration performed before the target model is trained can be called preliminary calibration, which can ensure the accuracy of subsequent training. After performing preliminary calibration (accuracy alignment) on the target model, embodiments of this disclosure can instruct each heterogeneous chip to train the target chip.

[0115] In step S240, multiple heterogeneous chips are instructed to collaboratively train the target model.

[0116] For different hardware platforms (heterogeneous chips), this embodiment of the disclosure can perform real-time accuracy compensation on the calculation results based on error characteristics, thereby ensuring that the output results remain consistent during training. That is, after performing preliminary calibration, this embodiment of the disclosure can instruct each heterogeneous chip to start collaboratively training the target model, and can also perform accuracy calibration operations in real time during the training of the target model.

[0117] As a specific implementation method, based on the accuracy detection report, this embodiment of the disclosure can perform real-time correction of computational errors generated by the chip during training. In other words, during the training of the target model, the main control device can apply an accuracy compensation mechanism to the output results of each chip based on the accuracy detection report (error feature report).

[0118] Here, the compensation mechanism can be implemented by increasing the number of bits in the floating-point number, adjusting the rounding strategy, or using a higher precision numerical representation. For example, in floating-point arithmetic, if it is determined that the error range of a certain chip is large, the precision of the chip can be increased to 64-bit floating-point numbers to reduce the error.

[0119] As can be seen, the accuracy calibration in this embodiment is not static, but dynamic, which can address accuracy drift during training. As the target model training progresses, some chips may exhibit different error characteristics. In this case, this embodiment can recalculate the correction coefficients (errors) and apply new compensation strategies.

[0120] It should be noted that, during the calibration of the computational accuracy of each heterogeneous chip, the embodiments of this disclosure can also determine the critical path of the target model, and on this basis, align the accuracy corresponding to the critical path. Here, the critical path can be obtained through the above-mentioned static analysis of the computational graph, node attribute analysis, and critical operator identification.

[0121] In other words, in the case of multi-chip collaborative computing, the embodiments of this disclosure can prioritize the alignment of the precision of critical paths (such as the transmission and calculation of intermediate results) to ensure that each chip always uses the same precision standard during the operation, thus avoiding the cumulative error caused by inconsistent precision.

[0122] It should be noted that the main control device in this embodiment can access and control the computational precision parameters of each heterogeneous chip. For example, the main control chip can be a high-precision computing unit on an FPGA and a dedicated AI accelerator.

[0123] Furthermore, differences in the computational precision characteristics of different chips may affect training efficiency. Therefore, when performing precision calibration operations, embodiments of this disclosure can balance the relationship between precision alignment and computational efficiency. For example, when training the 7th network layer of the target model using a first heterogeneous chip, if the active device determines that the operating error exceeds a preset error, the computational efficiency of the first heterogeneous chip can be obtained. If it is determined that the computational efficiency of the first heterogeneous chip exceeds the preset efficiency, embodiments of this disclosure can continue to use the first heterogeneous chip to train the 7th network layer of the target model. Conversely, if it is determined that the computational efficiency of the first heterogeneous chip does not exceed the preset efficiency, embodiments of this disclosure can instruct a second heterogeneous chip to train the 7th network layer of the target model.

[0124] The aforementioned accuracy calibration can be performed based on the accuracy test report. That is, in this embodiment of the disclosure, the error information corresponding to each heterogeneous chip in the accuracy test report can be used to perform accuracy calibration on the heterogeneous chips in real time. In addition, based on the accuracy calibration, this embodiment of the disclosure can also periodically perform consistency checks on the calculation results of each heterogeneous chip to ensure the consistency of calculations by the heterogeneous chips during the training process. For details, please refer to steps S250 to S270.

[0125] In step S250, intermediate computation results of each heterogeneous chip are collected for training the target model.

[0126] As an optional approach, during the collaborative training of the target model by the heterogeneous chips, this embodiment of the disclosure can collect the intermediate computation results of each heterogeneous chip. Specifically, the main control device can periodically sample the intermediate computation results of each heterogeneous chip, such as activation values ​​and gradient values. Based on this, this embodiment of the disclosure can compare the computation results between the heterogeneous chips respectively to obtain the comparison results, i.e., proceed to step S260.

[0127] In step S260, the intermediate calculation results of multiple heterogeneous chips are compared to obtain a comparison result.

[0128] As an optional approach, after obtaining the intermediate calculation results for each heterogeneous chip, embodiments of this disclosure can compare these intermediate calculation results to obtain a comparison result, thereby ensuring that the output results of each heterogeneous chip are consistent in the same task. Optionally, embodiments of this disclosure can also compare the intermediate calculation results with preset standard results.

[0129] In step S270, a consistency check is performed on multiple heterogeneous chips based on the comparison results.

[0130] In this embodiment of the disclosure, the comparison result of the intermediate calculation results of multiple heterogeneous chips can be the difference between the intermediate calculation result of each heterogeneous chip and the average calculation result, or it can be the difference between the current heterogeneous chip and each heterogeneous chip. For example, the accuracy calibration system includes a first heterogeneous chip, a second heterogeneous chip, and a third heterogeneous chip. Through training, the intermediate calculation result 1 of the first heterogeneous chip, the intermediate calculation result 2 of the second heterogeneous chip, and the intermediate calculation result 3 of the third heterogeneous chip can be obtained.

[0131] When calculating the error between the first heterogeneous chip and other chips, embodiments of this disclosure can obtain the difference between intermediate calculation result 1 and intermediate calculation result 2, and the difference between intermediate calculation result 1 and intermediate calculation result 3. Both of these differences can be used as the comparison result of the first heterogeneous chip. Optionally, the comparison result of the first heterogeneous chip can also be the difference between intermediate calculation result 1 and the average calculation result, wherein the average calculation result can be the average of intermediate calculation result 1, intermediate calculation result 2, and intermediate calculation result 3.

[0132] Based on this, embodiments of this disclosure can perform consistency checks on multiple heterogeneous chips according to the comparison results. Specifically, it determines whether the comparison results exceed a preset range; if they do, the calibration of the heterogeneous chip is strengthened. For example, if it is determined that the comparison result of the first heterogeneous chip exceeds the preset range, the calibration intensity of the first heterogeneous chip can be increased.

[0133] Specifically, embodiments of this disclosure can determine whether there is a first chip among multiple heterogeneous chips whose comparison result exceeds a preset error. If a first chip is determined to exist, the training of the target model is terminated, and the computational accuracy of the first chip is recalibrated. Here, the comparison result can be a cumulative error, meaning that embodiments of this disclosure can continuously monitor the cumulative errors that may be generated by different heterogeneous chips during the training process. If it is determined that the cumulative error exceeds a preset range, the training of the target model can be terminated, and error correction can be performed.

[0134] As an optional approach, when a first chip is identified among multiple heterogeneous chips, embodiments of this disclosure can determine that the first chip is performing a specified task. If it is determined that the first chip is performing a specified task, embodiments of this disclosure can increase the accuracy correction strength of the first chip. That is, when it is found that some chips have excessively large errors in a specific computing task, embodiments of this disclosure can increase the accuracy correction strength of those chips.

[0135] As an alternative approach, when it is determined that a first chip exists among multiple heterogeneous chips, and that the first chip is performing a specified task, embodiments of this disclosure can adjust the allocation method of the specified task to instruct a second chip to perform the specified task. Here, the computational precision of the second chip can be higher than that of the first chip. By adjusting the task allocation method, embodiments of this disclosure can preferentially assign tasks to chips with higher precision.

[0136] Based on the results of the consistency check, the embodiments of this disclosure can dynamically adjust the accuracy correction mechanism to ensure that the error during training remains within a controllable range. It should be noted that the consistency check can be performed based on an offline sample set, or it can be implemented during actual training by inserting consistency checkpoints.

[0137] Furthermore, during the consistency verification process, this embodiment of the disclosure can consider the time and space complexity of the computation to avoid system performance degradation due to frequent verifications. That is, during the consistency verification process, this embodiment of the disclosure can obtain the time and space complexity of the computation; if the time and space complexity meet the conditions, the consistency verification continues.

[0138] Conversely, if the time and space complexity do not meet the conditions, the number of consistency checks is reduced. Here, the conditions can be stated as either the time complexity exceeding a time complexity threshold or the space complexity exceeding a space complexity threshold.

[0139] During the training of the target model, the accuracy of collaborative computation across heterogeneous chips can remain consistent, thus ensuring the normal progress of the target model training. By maintaining consistent computational accuracy across different hardware platforms (heterogeneous chips), it can be ensured that regardless of which chip the task is assigned to during training, the intermediate output results and the final model will not be affected by accuracy errors.

[0140] As described above, consistency checks can be performed during model training. By comparing the computational results of different heterogeneous chips in the same task, the errors between the heterogeneous chips can be controlled within a preset range.

[0141] It should be noted that the consistency check is not limited to comparing the results of a single calculation, but can also include checking the cumulative error during continuous calculations. This can prevent the accumulation of errors from causing the target model training to fail. In addition, if the calculation results of some heterogeneous chips are detected to deviate from the preset error range during the consistency check, the embodiments of this disclosure can readjust the accuracy correction algorithm to further reduce the error.

[0142] In summary, throughout the model training process, the embodiments of this disclosure can dynamically monitor the accuracy performance of each heterogeneous chip and adjust the correction algorithm in real time. Furthermore, the main control device can adaptively adjust according to the progress of the target model training to ensure accuracy alignment throughout the entire training process. The dynamic monitoring mechanism can detect potential accuracy deviations that may occur in the chips during long-term operation and promptly compensate for them, preventing error accumulation.

[0143] This disclosed embodiment improves the accuracy of target model training by calibrating the computational precision of multiple heterogeneous chips. Specifically, a precision test command is sent to each heterogeneous chip, instructing it to run the target model. Based on this, the computational results from each heterogeneous chip are received and compared with standard computation results to obtain a precision detection report. This report includes precision error information for each heterogeneous chip. Finally, based on this report, the computational precision of multiple heterogeneous chips can be accurately calibrated. Since the precision detection report is obtained by running the target model on each heterogeneous chip, chip precision alignment can be achieved more accurately and efficiently. Furthermore, this patent disclosure embodiment, through precision detection reports, dynamic precision compensation, and automated data embedding strategies, can efficiently, accurately, and automatically achieve heterogeneous chip precision detection and calibration, making it applicable to precision debugging and optimization in multi-chip, multi-operator environments.

[0144] Figure 7 This is a block diagram illustrating a precision calibration apparatus between heterogeneous chips according to an exemplary embodiment. (Refer to...) Figure 7The precision calibration device 300 between heterogeneous chips includes a sending module 310, a generating module 320, and a calibration module 330.

[0145] The sending module 310 is configured to send a precision test command to each of the heterogeneous chips, the precision test command being used to instruct each of the heterogeneous chips to run the target model;

[0146] The generation module 320 is configured to receive the calculation results for the target model sent by each of the heterogeneous chips, compare the calculation results with the standard calculation results, and obtain an accuracy detection report, the accuracy detection report including the accuracy error information of each of the heterogeneous chips;

[0147] The calibration module 330 is configured to calibrate the computational accuracy of the plurality of heterogeneous chips based on the accuracy detection report.

[0148] In some embodiments, the precision calibration device 300 between heterogeneous chips further includes:

[0149] The training module is configured to instruct the multiple heterogeneous chips to collaboratively train the target model;

[0150] The acquisition module is configured to acquire intermediate computation results for each of the heterogeneous chips during the training of the target model.

[0151] The comparison module is configured to compare intermediate calculation results of multiple heterogeneous chips to obtain a comparison result;

[0152] A consistency verification module is configured to perform consistency verification on multiple heterogeneous chips based on the comparison results.

[0153] In some implementations, the consistency verification module includes:

[0154] The determination submodule is configured to determine whether there is a first chip among the plurality of heterogeneous chips whose comparison result exceeds a preset error;

[0155] The recalibration submodule is configured to terminate the training of the target model and recalibrate the computational accuracy of the first chip if the presence of the first chip is determined.

[0156] In some implementations, the recalibration submodule is configured to increase the precision correction intensity of the first chip when it is determined that the first chip is present among the plurality of heterogeneous chips and the first chip is performing a specified task.

[0157] In some implementations, the consistency verification module further includes:

[0158] The adjustment submodule is configured to, when it is determined that the first chip exists among the plurality of heterogeneous chips, if it is determined that the first chip is performing a specified task, adjust the allocation method of the specified task and instruct the second chip to perform the specified task, wherein the calculation accuracy of the second chip is higher than that of the first chip.

[0159] In some embodiments, the precision calibration device 300 between heterogeneous chips further includes:

[0160] The computational graph analysis module is configured to perform computational graph analysis on the target model and obtain computational graph analysis results.

[0161] The identification module is configured to perform node attribute analysis and key operator identification based on the computation graph analysis results to obtain critical path analysis results.

[0162] The event tracking insertion module is configured to generate an event tracking strategy based on the critical path analysis results, and to insert event tracking points in the target model based on the event tracking strategy.

[0163] In some implementations, the generation module 320 is configured to receive the calculation results of the target model at the embedded points sent by each of the heterogeneous chips; obtain the error between the calculation results and the standard calculation results; and write the error into the accuracy detection report.

[0164] In some implementations, the generation module 320 is further configured to mark the error when it is determined that the error exceeds a preset error, and to write the marked error into the accuracy detection report.

[0165] In some implementations, the generation module 320 is further configured to perform dynamic precision compensation on the calculation result when it is determined that the error exceeds a preset error.

[0166] This embodiment of the disclosure improves the accuracy of target model training by calibrating the computational precision of multiple heterogeneous chips. Specifically, a precision test command is sent to each heterogeneous chip to instruct it to run the target model. Based on this, the computational results for the target model sent by each heterogeneous chip are received and compared with standard computation results to obtain a precision detection report. The precision detection report includes the precision error information of each heterogeneous chip. Finally, based on the precision detection report, the computational precision of multiple heterogeneous chips can be accurately calibrated. Since the precision detection report is obtained by running the target model on each heterogeneous chip, chip precision alignment can be achieved more accurately and efficiently based on the report.

[0167] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0168] Figure 8 This is a block diagram illustrating an electronic device 700 according to an exemplary embodiment. Figure 8 As shown, the electronic device 700 may include a processor 701 and a memory 702. The electronic device 700 may also include one or more of a multimedia component 703, an input / output (I / O) interface 704, and a communication component 705.

[0169] The processor 701 controls the overall operation of the electronic device 700 to complete all or part of the steps in the aforementioned heterogeneous chip precision calibration method. The memory 702 stores various types of data to support the operation of the electronic device 700. This data may include, for example, instructions for any application or method operating on the electronic device 700, and application-related data such as contact data, sent and received messages, pictures, audio, video, etc. The memory 702 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read-Only Memory (EPROM), Programmable Read-Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The multimedia component 703 may include a screen and audio components. The screen may be, for example, a touchscreen, and the audio component is used to output and / or input audio signals. For example, the audio component may include a microphone for receiving external audio signals. The received audio signals may be further stored in memory 702 or transmitted via communication component 705. The audio component also includes at least one speaker for outputting audio signals. I / O interface 704 provides an interface between processor 701 and other interface modules, such as a keyboard, mouse, buttons, etc. These buttons may be virtual or physical buttons. Communication component 705 is used for wired or wireless communication between the electronic device 700 and other devices. Wireless communication may include Wi-Fi, Bluetooth, Near Field Communication (NFC), 2G, 3G, or 4G, or a combination thereof; therefore, the corresponding communication component 705 may include a Wi-Fi module, a Bluetooth module, or an NFC module.

[0170] In an exemplary embodiment, the electronic device 700 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the aforementioned precision calibration method between heterogeneous chips.

[0171] In another exemplary embodiment, a computer-readable storage medium including program instructions is also provided, which, when executed by a processor, implement the steps of the above-described precision calibration method between heterogeneous chips. For example, the computer-readable storage medium may be the memory 702 including program instructions, which may be executed by the processor 701 of the electronic device 700 to complete the above-described precision calibration method between heterogeneous chips.

[0172] In another exemplary embodiment, a computer program product is also provided, which includes a computer program executable by a processor, wherein the computer program, when executed by the processor, implements the steps of the above-described precision calibration method between heterogeneous chips.

[0173] Figure 9 This is a block diagram illustrating a server 1900 according to an exemplary embodiment. For example, server 1900 may be provided as a server. (Refer to...) Figure 9 The server 1900 includes processors 1922, which may be one or more, and memory 1932 for storing computer programs executable by the processors 1922. The computer programs stored in memory 1932 may include one or more modules, each corresponding to a set of instructions. Furthermore, the processor 1922 may be configured to execute the computer program to perform the aforementioned precision calibration method between heterogeneous chips.

[0174] Additionally, server 1900 may include a power supply component 1926 and a communication component 1950. The power supply component 1926 can be configured to perform power management for server 1900, and the communication component 1950 can be configured to enable communication for server 1900, such as wired or wireless communication. Furthermore, server 1900 may include an input / output (I / O) interface 1958. Server 1900 can operate on an operating system stored in memory 1932, such as Windows Server™, Mac OSX™, Unix™, Linux™, etc.

[0175] In another exemplary embodiment, a computer-readable storage medium including program instructions is also provided, which, when executed by a processor, implement the steps of the above-described precision calibration method between heterogeneous chips. For example, the computer-readable storage medium may be the memory 1932 including the program instructions, which may be executed by the processor 1922 of the server 1900 to complete the above-described precision calibration method between heterogeneous chips.

[0176] Those skilled in the art will also understand that the various illustrative logical blocks and steps listed in the embodiments of this application can be implemented by electronic hardware, computer software, or a combination of both. Whether such functionality is implemented through hardware or software depends on the specific application and the overall system design requirements. Those skilled in the art can implement the described functionality using various methods for each specific application, but such implementation should not be construed as exceeding the scope of protection of the embodiments of this application.

[0177] It should be understood that, unless otherwise specifically indicated, features of various embodiments of this disclosure described herein can be combined with each other. As used herein, the term “and / or” includes any one of the relevant listed items and any combination of any two or more; similarly, “at least one of…” includes any one of the relevant listed items and any combination of any two or more.

[0178] Although terms such as “first,” “second,” and “third” may be used herein to describe various components, parts, regions, layers, or sections, these components, parts, regions, layers, or sections are not limited to these terms. Rather, these terms are used only to distinguish one component, part, region, layer, or section from another. Therefore, without departing from the teachings of the examples described herein, the first component, part, region, layer, or section mentioned in the examples may also be referred to as the second component, part, region, layer, or section. Furthermore, the terms “first” and “second” are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined as “first” or “second” may explicitly or implicitly include at least one of that feature. In the description herein, “a plurality” means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0179] Furthermore, the term “exemplary” is used herein to mean serving as an example, instance, or illustration. Any aspect or design described herein as “exemplary” is not necessarily to be construed as advantageous compared to other aspects or designs. Rather, the use of the term “exemplary” is intended to present the concept in a concrete manner. As used herein, the term “or” is intended to mean an inclusive “or” rather than an exclusive “or.” That is, unless otherwise specified or clear from the context, “X applies A or B” is intended to mean any of the natural inclusive arrangements. That is, “X applies A or B” satisfies any of the foregoing instances if X applies A; X applies B; or both X applies A and B. Additionally, unless otherwise specified or clear from the context to refer to the singular form, the articles “a” and “an” as used in this application and the appended claims are generally understood to mean “one or more.”

[0180] Similarly, although this disclosure has been shown and described with respect to one or more implementations, equivalent variations and modifications will occur to those skilled in the art upon reading and understanding this specification and the accompanying drawings. This disclosure includes all such modifications and variations and is limited only by the scope of the claims. In particular, with respect to the various functions performed by the components described above (e.g., elements, resources, etc.), unless otherwise indicated, the terminology used to describe such components is intended to correspond to any component (functionally equivalent) that performs the specific function of the described component, even if structurally not equivalent to the disclosed structure. Furthermore, although specific features of this disclosure may have been disclosed with respect to only one of several implementations, such features may be combined with one or more other features of other implementations, as may be desired and advantageous to any given or particular application. Moreover, with regard to the terms “comprising,” “owning,” “having,” “having,” or variations thereof as used in the detailed description or claims, such terms are intended to be inclusive in a manner similar to the term “including.”

[0181] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the appended claims.

[0182] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. A method of precision calibration between heterogeneous chips, characterized by, The method is applied to a precision calibration system including a plurality of heterogeneous chips, and comprises the following steps: precision test instructions are respectively sent to each of the heterogeneous chips, the precision test instructions being used to instruct each of the heterogeneous chips to run a target model; operation results of the target model sent by each of the heterogeneous chips are received, and the operation results are compared with standard operation results to obtain a precision detection report, the precision detection report including precision error information of each of the heterogeneous chips; the calculation precision of the plurality of heterogeneous chips is calibrated based on the precision detection report; the plurality of heterogeneous chips are instructed to cooperatively train the target model; for training of the target model, intermediate calculation results of each of the heterogeneous chips are collected; the intermediate calculation results of the plurality of heterogeneous chips are compared to obtain a comparison result; consistency of the plurality of heterogeneous chips is verified according to the comparison result.

2. The precision calibration method between heterogeneous chips according to claim 1, wherein, The consistency of the plurality of heterogeneous chips is verified according to the comparison result, which comprises the following steps: it is determined whether there is a first chip in the plurality of heterogeneous chips whose comparison result exceeds a preset error; if it is determined that the first chip exists, the training of the target model is terminated, and the calculation precision of the first chip is recalibrated.

3. The precision calibration method between heterogeneous chips according to claim 2, wherein, If it is determined that the first chip exists, the training of the target model is terminated, and the calculation precision of the first chip is recalibrated, which comprises the following steps: when it is determined that the first chip exists in the plurality of heterogeneous chips, if it is determined that the first chip executes a specified task, the precision correction intensity of the first chip is increased.

4. The precision calibration method between heterogeneous chips according to claim 3, characterized by, The method further comprises the following steps: when it is determined that the first chip exists in the plurality of heterogeneous chips, if it is determined that the first chip executes a specified task, the allocation mode of the specified task is adjusted, and a second chip is instructed to execute the specified task, the calculation precision of the second chip being higher than that of the first chip.

5. The precision calibration method between heterogeneous chips according to claim 1, characterized by, Before the precision test instructions are respectively sent to each of the heterogeneous chips, the following steps are included: a computation graph analysis is performed on the target model to obtain a computation graph analysis result; node attribute analysis and key operator identification are performed according to the computation graph analysis result to obtain a critical path analysis result; a point-injection strategy is generated according to the critical path analysis result, and the point-injection strategy is used to insert a point-injection in the target model.

6. The precision calibration method between heterogeneous chips according to claim 5, wherein, The operation results of the target model sent by each of the heterogeneous chips are received, and the operation results are compared with standard operation results to obtain a precision detection report, which comprises the following steps: operation results of the target model at the point-injection sent by each of the heterogeneous chips are received; an error between the operation results and the standard operation results is obtained, and the error is written into the precision detection report.

7. The precision calibration method between heterogeneous chips according to claim 6, wherein, The error is written into the precision detection report, which comprises the following steps: when it is determined that the error exceeds a preset error, the error is marked, and the marked error is written into the precision detection report.

8. The precision calibration method between heterogeneous chips according to claim 7, wherein, The method further comprises the following steps: when it is determined that the error exceeds a preset error, dynamic precision compensation is performed on the operation results.

9. An apparatus for precision calibration between heterogeneous chips, comprising: The application is applied to a precision calibration system including a plurality of heterogeneous chips, and the device comprises: a sending module configured to send precision test instructions to each of the heterogeneous chips respectively, the precision test instructions being used to instruct each of the heterogeneous chips to run a target model; a generating module configured to receive operation results of the target model sent by each of the heterogeneous chips, and compare the operation results with standard operation results to obtain a precision detection report, the precision detection report including precision error information of each of the heterogeneous chips; a calibration module configured to calibrate calculation precision of the plurality of heterogeneous chips based on the precision detection report; a training module configured to instruct the plurality of heterogeneous chips to cooperatively train the target model; a collecting module configured to collect intermediate calculation results of each of the heterogeneous chips for training of the target model; a comparing module configured to compare the intermediate calculation results of the plurality of heterogeneous chips to obtain a comparison result; a consistency checking module configured to perform consistency checking on the plurality of heterogeneous chips according to the comparison result.

10. A computer-readable storage medium having stored thereon a computer program, characterized in that, The program is executed by the processor to implement the steps of the method in any one of claims 1-8.

11. An electronic device, comprising: comprise: a memory having a computer program stored thereon; a processor configured to execute the computer program in the memory to implement the steps of the method in any one of claims 1-8.

12. A computer program product comprising a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the method in any one of claims 1-8.

Images