Precision calibration method and device between heterogeneous chips, electronic equipment and medium

Abstract

The invention relates to a precision calibration method and device between heterogeneous chips, electronic equipment and a medium, the method and device are applied to a precision calibration system, the precision calibration system comprises a plurality of heterogeneous chips, and the method comprises the following steps: sending a precision test instruction to each heterogeneous chip, the precision test instruction is used for indicating each heterogeneous chip to operate a target model; an operation result for the target model sent by each heterogeneous chip is received, the operation result is compared with a standard operation result, a precision detection report is obtained, and the precision detection report comprises precision error information of each heterogeneous chip; and calibrating the calculation precision of the plurality of heterogeneous chips based on the precision detection report. According to the invention, the calculation precision of each heterogeneous chip can be accurately adjusted by using the generated precision detection report, so that the accuracy of model training can be ensured.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, and in particular, to a method, device, electronic device and medium for precision calibration between heterogeneous chips. Background Art

[0002] With the development of artificial intelligence (AI), the scale and complexity of AI models continue to increase, and the computing resources required for AI model training also increase. In order to improve the efficiency and speed of AI model training, the use of heterogeneous systems for distributed training has gradually become a trend.

[0003] However, different hardware architectures have different computing capabilities and precision characteristics, which makes cross-hardware platform training prone to inconsistent performance and accuracy, thus affecting the reliability of model training results. Summary of the invention

[0004] In order to overcome the problems existing in the related art, the present invention provides a method, device, electronic device and medium for calibrating the accuracy between heterogeneous chips. According to a first aspect of an embodiment of the present disclosure, a method for precision calibration between heterogeneous chips is provided, which is applied to a precision calibration system, wherein the precision calibration system includes a plurality of heterogeneous chips, and the method includes: Sending an accuracy test instruction to each of the heterogeneous chips respectively, wherein the accuracy test instruction is used to instruct each of the heterogeneous chips to run a target model; Receiving a calculation result for the target model sent by each of the heterogeneous chips, and comparing the calculation result with a standard calculation result to obtain an accuracy detection report, wherein the accuracy detection report includes accuracy error information of each of the heterogeneous chips; The calculation accuracy of the plurality of heterogeneous chips is calibrated based on the accuracy detection report.

[0005] Optionally, after calibrating the calculation accuracy of the plurality of heterogeneous chips based on the accuracy detection report, the method further comprises: Instructing the multiple heterogeneous chips to collaboratively train the target model; For the training of the target model, collecting the intermediate calculation results of each of the heterogeneous chips; Comparing the intermediate calculation results of the plurality of heterogeneous chips to obtain a comparison result; A consistency check is performed on the plurality of heterogeneous chips according to the comparison result.

[0006] Optionally, performing consistency check on the plurality of heterogeneous chips according to the comparison result includes: Determine whether there is a first chip among the plurality of heterogeneous chips whose comparison result exceeds a preset error; If it is determined that the first chip exists, the training of the target model is terminated, and the calculation accuracy of the first chip is recalibrated.

[0007] Optionally, if it is determined that the first chip exists, terminating the training of the target model and recalibrating the calculation accuracy of the first chip includes: When it is determined that the first chip exists among the plurality of heterogeneous chips, if it is determined that the first chip performs a designated task, the precision correction strength of the first chip is increased.

[0008] Optionally, when it is determined that the first chip exists among the multiple heterogeneous chips, if it is determined that the first chip performs a specified task, the allocation method of the specified task is adjusted to instruct the second chip to perform the specified task, and the calculation accuracy of the second chip is higher than that of the first chip.

[0009] Optionally, before sending the accuracy test instruction to each of the heterogeneous chips respectively, the method includes: Performing a computational graph analysis on the target model to obtain a computational graph analysis result; Perform node attribute analysis and key operator identification according to the calculation graph analysis results to obtain key path analysis results; A tracking strategy is generated according to the critical path analysis result, and tracking points are inserted into the target model based on the tracking strategy.

[0010] Optionally, the receiving a calculation result for the target model sent by each of the heterogeneous chips, and comparing the calculation result with a standard calculation result to obtain an accuracy detection report includes: Receiving a calculation result of the target model at the embedding point sent by each of the heterogeneous chips; Obtain an error between the calculation result and the standard calculation result, and write the difference error into the accuracy detection report.

[0011] Optionally, writing the error into the accuracy detection report includes: When it is determined that the error exceeds the preset error, the error is marked, and the marked error is written into the accuracy detection report.

[0012] Optionally, the method further comprises: When it is determined that the error exceeds a preset error, dynamic accuracy compensation is performed on the calculation result.

[0013] According to a second aspect of an embodiment of the present disclosure, there is provided a precision calibration device between heterogeneous chips, which is applied to a precision calibration system. The precision calibration system includes a plurality of heterogeneous chips. The device includes: A sending module is configured to send an accuracy test instruction to each of the heterogeneous chips respectively, wherein the accuracy test instruction is used to instruct each of the heterogeneous chips to run a target model; A generation module is configured to receive a calculation result for the target model sent by each of the heterogeneous chips, and compare the calculation result with a standard calculation result to obtain an accuracy detection report, wherein the accuracy detection report includes accuracy error information of each of the heterogeneous chips; The calibration module is configured to calibrate the calculation accuracy of the plurality of heterogeneous chips based on the accuracy detection report.

[0014] According to a third aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided, on which computer program instructions are stored. When the program instructions are executed by a processor, the steps of the accuracy calibration method between heterogeneous chips provided in the first aspect of the present disclosure are implemented.

[0015] According to a fourth aspect of an embodiment of the present disclosure, there is provided an electronic device, including: a memory having a computer program stored thereon; A processor is used to execute the computer program in the memory to implement the steps of the accuracy calibration method between heterogeneous chips provided in the first aspect of the present disclosure.

[0016] According to a fifth aspect of an embodiment of the present disclosure, a computer program product is provided, comprising a computer program, which, when executed by a processor, implements the steps of the method for precision calibration between heterogeneous chips provided in the first aspect of the present disclosure.

[0017] The present disclosure can improve the accuracy of target model training by calibrating the computing accuracy of multiple heterogeneous chips. Specifically, an accuracy test instruction is sent to each heterogeneous chip respectively, and the accuracy test instruction is used to instruct each heterogeneous chip to run the target model. On this basis, the operation result for the target model sent by each heterogeneous chip is received, and the operation result is compared with the standard operation result to obtain an accuracy detection report, wherein the accuracy detection report includes the accuracy error information of each heterogeneous chip. Finally, based on the accuracy detection report, the computing accuracy of multiple heterogeneous chips can be accurately calibrated. Since the accuracy detection report is obtained by letting each heterogeneous chip run the target model, the accuracy alignment of the chips can be achieved more accurately and efficiently based on the report.

[0018] Other features and advantages of the present disclosure will be described in detail in the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The accompanying drawings are used to provide a further understanding of the present disclosure and constitute a part of the specification. Together with the following specific embodiments, they are used to explain the present disclosure but do not constitute a limitation of the present disclosure. In the accompanying drawings: Figure 1 The present invention is a flowchart of a method for calibrating the accuracy between heterogeneous chips according to an exemplary embodiment.

[0020] Figure 2 The figure is a flow chart showing another method for calibrating accuracy between heterogeneous chips according to an exemplary embodiment.

[0021] Figure 3 The present invention is a flowchart of generating an accuracy detection report in another accuracy calibration method between heterogeneous chips according to an exemplary embodiment.

[0022] Figure 4 The diagram is a structural example diagram of a target model in another method for precision calibration between heterogeneous chips according to an exemplary embodiment.

[0023] Figure 5 It is an example diagram of a calculation graph corresponding to a target model in another method for precision calibration between heterogeneous chips according to an exemplary embodiment.

[0024] Figure 6 1 is an example diagram of accuracy outlier points in another accuracy calibration method between heterogeneous chips according to an exemplary embodiment.

[0025] Figure 7 The invention is a block diagram of a precision calibration device between heterogeneous chips according to an exemplary embodiment.

[0026] Figure 8 It is a block diagram of an electronic device according to an exemplary embodiment.

[0027] Fig. 9 It is a block diagram of a server according to an exemplary embodiment. DETAILED DESCRIPTION

[0028] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0029] In the description of the present disclosure, the terms used, such as "first", "second", etc., are used to distinguish similar objects and do not have to be understood as a specific order or sequence. In addition, in the description with reference to the accompanying drawings, the same symbols in different drawings represent the same elements unless otherwise stated.

[0030] Although operations or steps are described in a specific order in the drawings in the embodiments of the present disclosure, it should not be understood that it is required to perform these operations or steps in the specific order shown or in a serial order, or to perform all the operations or steps shown to obtain the desired results. In the embodiments of the present disclosure, these operations or steps can be performed in series; these operations or steps can also be performed in parallel; or some of these operations or steps can be performed.

[0031] From the above introduction, we know that in order to improve training efficiency and speed, more and more computing architectures use heterogeneous systems for distributed training. Heterogeneous systems can include CPU (Central Processing Unit), GPU (Graphics Processing Unit), FPGA (Field-Programmable Gate Array) and dedicated AI accelerators.

[0032] However, different hardware architectures have different computing capabilities and precision characteristics, especially in floating-point and integer arithmetic precision, which may be quite different. These differences can easily lead to inconsistencies in model performance and precision in cross-hardware platform training tasks, thus affecting the reliability of training results.

[0033] In related technologies, although heterogeneous computing architecture can optimize the utilization efficiency of different computing resources through task allocation, data parallelism, etc., how to ensure the consistency of computing results between multiple hardware platforms is still an urgent problem to be solved. For example, when performing floating-point calculations, some hardware may introduce errors due to precision limitations. If such errors cannot be corrected in time, they may accumulate during the training process, resulting in a decrease in the performance of the model obtained by the final training.

[0034] In addition, related technologies mainly improve the training efficiency of heterogeneous systems by optimizing time and energy consumption, and rarely involve how to effectively correct the precision errors across hardware platforms. That is, there is currently no effective means to solve the problem of computing precision alignment between heterogeneous chips. Especially when multiple hardware platforms collaborate to train AI models, there is still a lot of room for improvement in how to effectively control and correct the precision errors between different chips.

[0035] In order to solve the above problems, the embodiments of the present disclosure propose a method for precision calibration between heterogeneous chips. The method calibrates the computing precision of multiple heterogeneous chips by using a pre-generated precision detection report, thereby effectively correcting the precision errors between different chips, thereby improving the accuracy of artificial intelligence model training.

[0036] Figure 1 is a flow chart of a method for calibrating accuracy between heterogeneous chips according to an exemplary embodiment. Figure 1 As shown, the method may include the following steps.

[0037] In step S110, an accuracy test instruction is sent to each heterogeneous chip respectively.

[0038] The disclosed embodiments may be applied to an accuracy calibration system, which may include multiple heterogeneous chips that can collaborate to train a target model. Exemplarily, the accuracy calibration system may include a first heterogeneous chip and a second heterogeneous chip, wherein the first heterogeneous chip is used to train computing network layers 7 to 12 of the target model, and the second heterogeneous chip is used to train computing network layers 1 to 6 of the target model.

[0039] In the embodiment of the present disclosure, in addition to including multiple heterogeneous chips, the precision calibration system may also include a main control device, which may be an electronic device or a server. Here, the main control device may be used to implement precision calibration between heterogeneous chips, and specifically, the main control device may be used to send precision test instructions to each heterogeneous chip.

[0040] As an optional method, before generating an accuracy test report or before training a target model, the embodiment of the present disclosure may send an accuracy test instruction to each heterogeneous chip to instruct each heterogeneous chip to run the target model respectively through the accuracy test instruction.

[0041] Among them, the accuracy test instruction can be manually triggered by the user, or it can be triggered when the accuracy calibration system meets the preset conditions. For example, when the accuracy calibration system receives a scheduled task, it triggers the sending of the accuracy test instruction. For another example, when the accuracy calibration system receives a model training instruction or detects a training data collection instruction, it triggers the sending of the accuracy test instruction. There is no clear restriction on how to trigger the accuracy test instruction here, and it can be selected according to the actual situation.

[0042] In some implementations, the precision test instruction is used to analyze the calculation precision of each heterogeneous chip. Specifically, the precision test instruction can be used to analyze at least one calculation precision of each heterogeneous chip, including floating point calculation precision, integer calculation precision, rounding error calculation precision, and numerical overflow calculation precision.

[0043] As an optional method, the accuracy test instruction may include a program for testing, and the program may be a program related to the target model, that is, the accuracy test instruction may include the target model, and by sending the target model to each heterogeneous chip, each heterogeneous chip may be instructed to execute the target model respectively to obtain an operation result. Here, the target model operation may involve addition, multiplication, matrix product, dot product, etc.

[0044] Optionally, the precision test instruction may further include standard operator operations, and by sending the standard operator operations to each heterogeneous chip, each heterogeneous chip may perform a series of standard operator operations. Here, the standard operator operations may include addition, multiplication, matrix product, etc.

[0045] Optionally, the precision test instruction may also include an operation acquisition address, that is, when the precision test instruction does not include a specified operation, it may be determined whether the instruction carries an address storing the specified operation, and the specified operation may be found based on the address. On this basis, the heterogeneous chip may execute the specified operation to obtain the operation result.

[0046] After receiving the accuracy test instruction, each heterogeneous chip in the accuracy calibration system can execute the operation in the accuracy test instruction, such as running the specified operation indicated by the heterogeneous chip, which can be a target model or a standard operator operation.

[0047] It should be noted that the operations received by each heterogeneous chip can be the same, that is, the embodiment of the present disclosure can instruct multiple heterogeneous chips to perform the same operation at the same time and obtain multiple operation results, so that a more accurate accuracy detection report can be obtained.

[0048] In step S120, the operation results for the target model sent by each heterogeneous chip are received, and the operation results are compared with the standard operation results to obtain an accuracy detection report.

[0049] As an optional method, after sending the accuracy test instruction to each heterogeneous chip, the embodiment of the present disclosure can detect whether the operation result for the target model sent by each heterogeneous chip is received. After receiving the operation result for the target model sent by each heterogeneous chip, the embodiment of the present disclosure can compare the operation result of each heterogeneous chip with the standard operation result to obtain an accuracy detection report.

[0050] In other words, the embodiments of the present disclosure can compare the intermediate calculation results of multiple heterogeneous chips with the standard calculation results respectively, and can obtain multiple comparison results, which can be accuracy error information.

[0051] Here, the standard operation result may be a preset operation result, which may be a result obtained by running the target model on a standard device. For example, the embodiment of the present disclosure may obtain correct data by running the same code on the NV / CPU, and the correct data may be used as the standard operation result.

[0052] In the embodiment of the present disclosure, the accuracy detection report may include the accuracy error information of each heterogeneous chip, that is, the accuracy error information may correspond to the heterogeneous chip. In addition, the same heterogeneous chip may correspond to multiple sub-error information, and the multiple sub-error information may include sub-error information of addition operation, sub-error information of multiplication operation, or sub-error information of matrix product operation.

[0053] In order to better illustrate the content of the accuracy test report, the present disclosure embodiment provides the information shown in Table 1 below: Table 1

[0054] The above calculation errors are only examples and are not intended to be actual limitations. The actual situation shall prevail.

[0055] In the disclosed embodiments, whether it is addition operation error, multiplication operation error or matrix product operation, etc., they are all obtained by comparing the calculation result of the heterogeneous chip with the standard operation result. For example, when chip 1 calculates the sum of two floating point numbers 1.00000001 and 0.99999999, the calculation result obtained is 2.00000001, and the standard operation result is 2.00000000. The error between the two is 2.00000001 − 2.00000000 = 0.00000001, that is, the addition operation error of chip 1 is 0.00000001.

[0056] By generating a hardware accuracy report (accuracy detection report), the differences in computing accuracy between various chips can be clearly presented, which can significantly improve the transparency of accuracy alignment and thus improve the efficiency of accuracy detection.

[0057] In step S130 , the calculation accuracy of the plurality of heterogeneous chips is calibrated based on the accuracy detection report.

[0058] As an optional method, after generating the accuracy detection report, the embodiment of the present disclosure can calibrate the calculation accuracy of multiple heterogeneous chips based on the accuracy detection report. Specifically, the actual operation result is compensated according to the accuracy error information corresponding to each heterogeneous chip. For example, when chip 1 is training the target model and performing an addition operation, the operation result obtained in real time can be compensated according to the addition operation error in the accuracy detection report.

[0059] By testing the computational accuracy of different hardware architectures (heterogeneous chips), the disclosed embodiments can obtain the computational error of each hardware architecture and perform targeted calibration, thereby ensuring the consistency of collaborative training results of different hardware architectures.

[0060] In a specific embodiment, the calculation accuracy calibration method may include but is not limited to increasing the floating point digits, adjusting the rounding strategy, or applying a high-precision computing unit, etc. Through accuracy calibration, the calculation results of each heterogeneous chip can be kept consistent with the set accuracy target.

[0061] The disclosed embodiment can improve the accuracy of target model training by calibrating the computing accuracy of multiple heterogeneous chips. Specifically, an accuracy test instruction is sent to each heterogeneous chip respectively, and the accuracy test instruction is used to instruct each heterogeneous chip to run the target model. On this basis, the operation result for the target model sent by each heterogeneous chip is received, and the operation result is compared with the standard operation result to obtain an accuracy detection report, wherein the accuracy detection report includes the accuracy error information of each heterogeneous chip. Finally, based on the accuracy detection report, the computing accuracy of multiple heterogeneous chips can be accurately calibrated. Since the accuracy detection report is obtained by letting each heterogeneous chip run the target model, the accuracy alignment of the chips can be achieved more accurately and efficiently based on the report.

[0062] Figure 2 is a flowchart of another method for calibrating the accuracy between heterogeneous chips according to an exemplary embodiment. Figure 2 As shown, the method may include the following steps.

[0063] In step S210, an accuracy test instruction is sent to each heterogeneous chip respectively.

[0064] As an optional method, in the process of generating an accuracy detection report, the embodiment of the present disclosure can first perform an automatic tracking operation, collect the calculation results of each heterogeneous chip for the target model according to the inserted tracking points, and based on the calculation results, a more accurate accuracy detection report can be obtained.

[0065] Here, the specific generation of the accuracy test report can be as follows Figure 3 As shown, based on Figure 3 It can be seen that after obtaining the target model, the embodiment of the present disclosure can first perform static analysis on the computation graph of the target model. For example, the network structure of the target model is as follows: Figure 4 As shown, it can be seen that the target model at this time can include an input layer, a hidden layer and an output layer.

[0066] On this basis, the embodiment of the present disclosure can obtain Figure 4 The computational graph of the target model shown in Figure 5At this point, you can Figure 5 The calculation graph shown in the figure performs node attribute analysis and key operator identification to obtain an identification result. Based on the identification result, the embodiment of the present disclosure can generate a corresponding point embedding strategy. On this basis, the embedding is automatically inserted into the target model according to the point embedding strategy, that is, the embedding is inserted into the code of the target model.

[0067] It can be seen that the embodiment of the present disclosure can perform calculation graph analysis on the target model to obtain the calculation graph analysis result. According to the calculation graph analysis result, node attribute analysis and key operator identification can be performed to obtain the key path analysis result. On this basis, a point embedding strategy can be generated according to the key path analysis result, and a point embedding can be inserted into the target model based on the point embedding strategy. Among them, the point embedding can be used for the collection of subsequent calculation results.

[0068] It should be noted that the disclosed embodiment may insert the buried points into the target model during the process of sending the precision test instruction to each heterogeneous chip, or may send the precision test instruction to the target model and then instruct each heterogeneous chip to insert the buried points into the target model respectively. There is no clear restriction on when to insert the buried points into the target model, and it can be selected according to the actual situation.

[0069] In summary, the embodiments of the present disclosure can automatically identify key nodes and insert monitoring points by statically analyzing the computation graph, thereby automatically capturing and monitoring the accuracy and performance of model operators.

[0070] In step S220, the operation results for the target model sent by each heterogeneous chip are received, and the operation results are compared with the standard operation results to obtain an accuracy detection report.

[0071] In some implementations, after obtaining a target model containing buried points, each heterogeneous chip can run the target model, and in the process of running the target model, it can collect operation results based on the automatically inserted buried points. Through the automatic buried point and capturer insertion technology, the disclosed embodiment can reduce the workload of developers to manually insert monitoring points, thereby improving the automation and efficiency of operator precision detection and debugging.

[0072] At this time, the main control device can receive the calculation results collected at the buried points for the target model sent by each heterogeneous chip, and on this basis, obtain the error between each calculation result and the standard calculation result, and write the error into the accuracy detection report. Among them, the target model can automatically insert multiple buried points, so each heterogeneous chip can also send multiple calculation results, so there are also multiple corresponding errors. In the process of writing multiple errors into the accuracy detection report, the embodiment of the present disclosure can write the error to the corresponding position of the heterogeneous chip.

[0073] Exemplarily, the multiplication error and the addition error of the first heterogeneous chip can be written to the location where the first heterogeneous chip identifier 00 is located; the multiplication error and the addition error of the second heterogeneous chip can be written to the location where the second heterogeneous chip identifier 01 is located.

[0074] It should be noted that, in the process of writing the error between the calculation result and the standard calculation result into the accuracy detection report, the embodiment of the present disclosure can observe whether there is an accuracy anomaly, that is, observe the accuracy anomaly point.

[0075] Specifically, it is determined whether the error between the operation result and the standard operation result is greater than the preset error. If the error between the operation result and the standard operation result is greater than the preset error, the embodiment of the present disclosure can mark the error and write the marked error into the accuracy detection report. In other words, when an accuracy anomaly is observed, the embodiment of the present disclosure can highlight the accuracy anomaly code.

[0076] At this time, the embodiment of the present disclosure can perform dynamic accuracy compensation, that is, when it is determined that the error between the calculation result and the standard calculation result exceeds the preset error, the embodiment of the present disclosure can perform dynamic accuracy compensation on the calculation result. After that, the embodiment of the present disclosure can run the remaining models to the end, and record the accuracy outliers to obtain the final accuracy detection report. Here, the accuracy outliers can be as follows Figure 6 As shown, the points indicated by marker 601, marker 602 and marker 603 are all accuracy abnormal points, and the other points are accuracy normal points.

[0077] As an example, when running the 6th layer network of the target model, a buried point is found. At this time, the operation result at the buried point can be collected. If it is determined that the error between this operation result and the standard operation result exceeds the preset error, the buried point can be recorded, that is, the code corresponding to the buried point is marked as an accuracy anomaly point.

[0078] At the same time, the disclosed embodiment can perform dynamic precision compensation on the calculation results here to avoid error accumulation and affect the acquisition of subsequent calculation results. When encountering precision errors, the reference precision (standard calculation results, such as NV or CPU precision) is automatically used to compensate for subsequent calculations to avoid the accumulation of precision errors. The dynamic precision compensation mechanism can identify the precision errors of the entire network in one operation, avoid multiple debugging of the entire network, reduce the number of debugging times from O(n) to O(1), and significantly shorten the debugging time.

[0079] It should be noted that the accuracy detection report can be generated before the target model is trained. That is, before training the target model, the embodiment of the present disclosure can first analyze the computing accuracy of each heterogeneous chip. The specific analysis content includes floating-point operation accuracy, integer operation accuracy, rounding error, numerical overflow, etc.

[0080] By setting a standard test set, the disclosed embodiment can perform test calculations on each chip separately, and collect the deviation data of each chip in floating-point and integer numerical operations. On this basis, a precision feature analysis report (precision detection report) is generated according to the analysis results, which can be used to describe the error distribution and error range of each chip in different operations. In other words, by analyzing the basic calculation operators, the calculation performance and precision reports of operators on different chips can be obtained.

[0081] As a specific implementation, before training the target model, the disclosed embodiment can analyze and collect the accuracy characteristics of different heterogeneous chips. Specifically, a standard operator test operation is performed to instruct the heterogeneous chips to perform a series of standard operator operations (run the target model), such as matrix multiplication, dot product, etc., and collect the execution results (operation results) of these operators.

[0082] On this basis, error analysis is performed, and the collected chip execution results are compared with the standard precision results (standard calculation results), the error range of each chip in floating-point and integer operations is obtained, and the error analysis results are obtained. The error analysis results may include error ranges such as calculation accuracy, rounding error, and overflow.

[0083] Next, an accuracy feature report (accuracy detection report) is generated based on the error analysis results. The accuracy detection report may include the accuracy error range and calculation deviation of each operator on different chips. The accuracy detection report can be used as a reference for subsequent accuracy alignment.

[0084] In summary, the embodiments of the present disclosure can obtain a precision detection report by collecting precision features of heterogeneous chips. The process of generating the precision detection report can be based on a specific test set or labeled data (target model), and the operator tasks performed by each heterogeneous chip can be the same, thereby ensuring the accuracy of the precision features.

[0085] It should be noted that, in the process of generating the accuracy detection report, the embodiment of the present disclosure may consider the influence of the chip's operating frequency, temperature, running time, etc. on the accuracy, to ensure that the acquired accuracy features have long-term stability. Exemplarily, the embodiment of the present disclosure may instruct the heterogeneous chip to run the target model under an extreme environment, such as running the target model when the temperature exceeds the preset temperature, so as to avoid the influence of the extreme environment on the accuracy calibration.

[0086] It should also be noted that after obtaining the comparison result between the operation result and the standard operation result, the embodiment of the present disclosure can classify the precision error of each chip. By comparing the standard operation result with the calculation result output by the chip, the typical error on each operation (such as addition, multiplication, matrix product, etc.) can be determined. Here, the classification can include normal errors and abnormal errors. In addition, for operation operations with large precision errors, the embodiment of the present disclosure can record its error range and generate a precision detection report for subsequent precision correction.

[0087] In step S230, the calculation accuracy of the plurality of heterogeneous chips is calibrated based on the accuracy detection report.

[0088] As an optional method, after obtaining the accuracy detection report, the embodiment of the present disclosure can calibrate the computing accuracy of each heterogeneous chip based on the accuracy detection report. Here, the calibration of the computing accuracy can be performed before the target model is trained, or it can be adjusted in real time during the target model training process. The calibration performed before the target model training can be called a preliminary calibration, and the accuracy of subsequent training can be guaranteed by the preliminary calibration operation. After the target model is preliminarily calibrated (accuracy alignment), the embodiment of the present disclosure can instruct each heterogeneous chip to train the target chip.

[0089] In step S240, multiple heterogeneous chips are instructed to collaboratively train a target model.

[0090] For different hardware platforms (heterogeneous chips), the embodiments of the present disclosure can perform real-time accuracy compensation on the calculation results based on the error characteristics, thereby ensuring that the output results remain consistent during the training process. That is, after performing the preliminary calibration, the embodiments of the present disclosure can instruct each heterogeneous chip to start collaboratively training the target model, and during the training of the target model, the accuracy calibration operation can also be performed in real time.

[0091] As a specific implementation, according to the accuracy detection report, the disclosed embodiment can perform real-time correction on the calculation errors generated by the chip during the training process. In other words, during the target model training process, the master control device can apply the accuracy compensation mechanism to the output results of each chip according to the accuracy detection report (error characteristic report).

[0092] Here, the compensation mechanism can be implemented by increasing the number of bits of floating point numbers, adjusting the rounding strategy, or using higher-precision numerical representation. For example, in floating-point operations, if it is determined that the error range of a certain chip is large, the accuracy of the chip can be increased to 64-bit floating point numbers to reduce the error.

[0093] It can be seen that the accuracy calibration in the embodiment of the present disclosure is not static, but dynamic calibration, and the accuracy drift during the training process can be dealt with based on dynamic calibration. As the target model training progresses, some chips may exhibit different error characteristics. At this time, the embodiment of the present disclosure can recalculate the correction coefficient (error) and apply a new compensation strategy.

[0094] It should be noted that, in the process of calibrating the computational accuracy of each heterogeneous chip, the disclosed embodiment can also determine the critical path of the target model, and on this basis, align the accuracy corresponding to the critical path. Here, the critical path can be obtained through the above-mentioned computational graph static analysis, node attribute analysis, and key operator identification.

[0095] That is to say, in cases involving multi-chip collaborative computing, the embodiments of the present disclosure can prioritize the alignment of the precision of critical paths (such as the transmission and calculation of intermediate results) to ensure that each chip always uses the same precision standard during the calculation process, thereby avoiding cumulative errors caused by inconsistent precision.

[0096] It should be noted that the master control device in the embodiment of the present disclosure can access and adjust the calculation accuracy parameters of each heterogeneous chip. For example, the master control chip can perform high-precision computing units on FPGA and dedicated AI accelerators.

[0097] In addition, the differences in computing accuracy characteristics of different chips may affect the training efficiency, so the embodiment of the present disclosure can weigh the relationship between accuracy alignment and computing efficiency when performing the accuracy calibration operation. For example, when using the first heterogeneous chip to train the 7th network layer of the target model, the active device determines that the running error exceeds the preset error, and the computing efficiency of the first heterogeneous chip can be obtained at this time. If it is determined that the computing efficiency of the first heterogeneous chip exceeds the preset efficiency, the embodiment of the present disclosure can continue to use the first heterogeneous chip to train the 7th network layer of the target model. Conversely, if it is determined that the computing efficiency of the first heterogeneous chip does not exceed the preset efficiency, the embodiment of the present disclosure can instruct the second heterogeneous chip to train the 7th network layer of the target model.

[0098] The above-mentioned precision calibration can be performed on the basis of the precision detection report, that is, the embodiment of the present disclosure can use the error information corresponding to each heterogeneous chip in the precision detection report to perform precision calibration on the heterogeneous chips in real time. In addition, based on the precision calibration, the embodiment of the present disclosure can also regularly perform consistency checks on the calculation results of each heterogeneous chip to ensure the consistency of the calculation of the heterogeneous chips during the training process. For details, please refer to steps S250 to S270.

[0099] In step S250 , for the training of the target model, the intermediate calculation results of each heterogeneous chip are collected.

[0100] As an optional method, when the heterogeneous chips are collaborating to train the target model, the embodiment of the present disclosure can collect the intermediate calculation results of each heterogeneous chip. Specifically, the master control device can periodically sample the intermediate calculation results of each heterogeneous chip, such as activation values, gradient values, etc. On this basis, the embodiment of the present disclosure can compare the calculation results between the heterogeneous chips respectively to obtain the comparison results, that is, enter step S260.

[0101] In step S260, the intermediate calculation results of the plurality of heterogeneous chips are compared to obtain a comparison result.

[0102] As an optional method, after obtaining the intermediate calculation results of each heterogeneous chip, the embodiment of the present disclosure can compare these intermediate calculation results to obtain a comparison result, so as to ensure that the results output by each heterogeneous chip in the same task are consistent. Optionally, the embodiment of the present disclosure can also compare the intermediate calculation results with a preset standard result.

[0103] In step S270, a consistency check is performed on the plurality of heterogeneous chips according to the comparison result.

[0104] In the disclosed embodiment, the comparison result of the intermediate calculation results of multiple heterogeneous chips may be the difference between the intermediate calculation result of each heterogeneous chip and the average calculation result, or may be the difference between the current heterogeneous chip and each heterogeneous chip. For example, the precision calibration system includes a first heterogeneous chip, a second heterogeneous chip, and a third heterogeneous chip. Through training, the intermediate calculation result 1 of the first heterogeneous chip, the intermediate calculation result 2 of the second heterogeneous chip, and the intermediate calculation result 3 of the third heterogeneous chip can be obtained.

[0105] When calculating the error between the first heterogeneous chip and other chips, the embodiment of the present disclosure can obtain the difference between the intermediate calculation result 1 and the intermediate calculation result 2, and obtain the difference between the intermediate calculation result 1 and the intermediate calculation result 3, and both of these differences can be used as the comparison result of the first heterogeneous chip. Optionally, the comparison result of the first heterogeneous chip can also be the difference between the intermediate calculation result 1 and the average calculation result, where the average calculation result can be the average of the intermediate calculation result 1, the intermediate calculation result 2, and the intermediate calculation result 3.

[0106] On this basis, the embodiment of the present disclosure can perform consistency check on multiple heterogeneous chips according to the comparison result. Specifically, it is determined whether the comparison result exceeds the preset range. If it exceeds the preset range, the calibration of the heterogeneous chip is strengthened. For example, if it is determined that the comparison result of the first heterogeneous chip exceeds the preset range, the calibration of the first heterogeneous chip can be strengthened.

[0107] Specifically, the embodiment of the present disclosure can determine whether there is a first chip among multiple heterogeneous chips whose comparison result exceeds a preset error. If it is determined that there is a first chip, the training of the target model is terminated and the calculation accuracy of the first chip is recalibrated. Here, the comparison result can be a cumulative error, that is, the embodiment of the present disclosure can continuously monitor the cumulative error that may be generated by different heterogeneous chips during the training process. If it is determined that the cumulative error exceeds the preset range, the training of the target model can be terminated and error correction can be performed.

[0108] As an optional method, when it is determined that there is a first chip among multiple heterogeneous chips, the embodiment of the present disclosure can determine that the first chip is performing a specified task. If it is determined that the first chip is performing a specified task, the embodiment of the present disclosure can increase the precision correction strength of the first chip. In other words, when it is found that the error of some chips on a specific computing task is too large, the embodiment of the present disclosure can increase the precision correction strength of the chip.

[0109] As another optional method, when it is determined that there is a first chip among multiple heterogeneous chips, and it is determined that the first chip performs a specified task, the embodiment of the present disclosure can adjust the allocation method of the specified task to instruct the second chip to perform the specified task. Here, the calculation accuracy of the second chip can be higher than the calculation accuracy of the first chip. By adjusting the allocation method of the task, the embodiment of the present disclosure can give priority to handing over the task to the chip with higher accuracy for processing.

[0110] According to the results of the consistency check, the embodiment of the present disclosure can dynamically adjust the accuracy correction mechanism to ensure that the error in the training process is always within a controllable range. It should be noted that the consistency check can be verified based on an offline sample set, and can also be implemented by inserting a consistency checkpoint in actual training.

[0111] In addition, in the process of consistency checking, the embodiment of the present disclosure may consider the time and space complexity of the calculation to avoid system performance degradation due to frequent checks. That is, in the process of consistency checking, the embodiment of the present disclosure may obtain the time complexity and space complexity of the calculation, and if the time complexity and space complexity meet the conditions, the consistency check will continue.

[0112] On the contrary, if the time complexity and space complexity do not meet the conditions, the number of consistency checks is reduced. Here, the condition can be that the time complexity exceeds the time complexity threshold, or the space complexity exceeds the space complexity threshold.

[0113] During the training of the target model, the accuracy of the collaborative calculations of various heterogeneous chips can be kept consistent, thereby ensuring the normal progress of the target model training. By maintaining consistent calculation accuracy between different hardware platforms (heterogeneous chips), it can be ensured that no matter which chip the task is assigned to during the training process, the output intermediate results and final model will not be affected by accuracy errors.

[0114] From the above introduction, we know that consistency check can be performed during the model training process. By comparing the calculation results of different heterogeneous chips in the same task, the error between heterogeneous chips can be controlled within a preset range.

[0115] It should be noted that the consistency check is not limited to the comparison of single operation results, but can also include the check of accumulated errors in the continuous calculation process, so as to prevent the error accumulation from causing the target model training to fail. In addition, if the calculation results of some heterogeneous chips are detected to deviate from the preset error range during the consistency check, the embodiment of the present disclosure can readjust the precision correction algorithm to further reduce the error.

[0116] In summary, during the entire process of model training, the disclosed embodiment can dynamically monitor the accuracy performance of each heterogeneous chip and adjust the correction algorithm in real time. In addition, the main control device can make adaptive adjustments based on the progress of the target model training to ensure the accuracy alignment state throughout the training process. The dynamic monitoring mechanism can detect the accuracy deviation that may occur in the chip during long-term operation, and make compensation processing in time to prevent error accumulation.

[0117] The disclosed embodiment can improve the accuracy of target model training by calibrating the calculation accuracy of multiple heterogeneous chips. Specifically, an accuracy test instruction is sent to each heterogeneous chip, and the accuracy test instruction is used to instruct each heterogeneous chip to run the target model. On this basis, the operation result for the target model sent by each heterogeneous chip is received, and the operation result is compared with the standard operation result to obtain an accuracy detection report, wherein the accuracy detection report includes the accuracy error information of each heterogeneous chip. Finally, based on the accuracy detection report, the calculation accuracy of multiple heterogeneous chips can be accurately calibrated. Since the accuracy detection report is obtained by letting each heterogeneous chip run the target model, the accuracy alignment of the chip can be achieved more accurately and efficiently based on the report. In addition, the disclosed embodiment of the patent can efficiently, accurately and automatically realize heterogeneous chip accuracy detection and calibration through accuracy detection reports, dynamic accuracy compensation and automated point-burying strategies, which can be applied to accuracy debugging and optimization work in a multi-chip and multi-operator environment.

[0118] Figure 7 is a block diagram of a precision calibration device between heterogeneous chips according to an exemplary embodiment. Figure 7The precision calibration device 300 between heterogeneous chips includes a sending module 310, a generating module 320 and a calibration module 330.

[0119] The sending module 310 is configured to send an accuracy test instruction to each of the heterogeneous chips respectively, wherein the accuracy test instruction is used to instruct each of the heterogeneous chips to run a target model; The generation module 320 is configured to receive the operation result for the target model sent by each of the heterogeneous chips, and compare the operation result with the standard operation result to obtain an accuracy detection report, wherein the accuracy detection report includes the accuracy error information of each of the heterogeneous chips; The calibration module 330 is configured to calibrate the calculation accuracy of the plurality of heterogeneous chips based on the accuracy detection report.

[0120] In some implementations, the precision calibration device 300 between heterogeneous chips further includes: A training module, configured to instruct the plurality of heterogeneous chips to collaboratively train the target model; A collection module, configured to collect intermediate calculation results of each of the heterogeneous chips for training the target model; A comparison module, configured to compare the intermediate calculation results of the plurality of heterogeneous chips to obtain a comparison result; The consistency check module is configured to perform a consistency check on the plurality of heterogeneous chips according to the comparison result.

[0121] In some implementations, the consistency check module includes: A determination submodule, configured to determine whether there is a first chip among the plurality of heterogeneous chips whose comparison result exceeds a preset error; The recalibration submodule is configured to terminate the training of the target model and recalibrate the calculation accuracy of the first chip if it is determined that the first chip exists.

[0122] In some implementations, the recalibration submodule is configured to, when it is determined that the first chip exists among the plurality of heterogeneous chips, increase the accuracy correction strength of the first chip if it is determined that the first chip performs a designated task.

[0123] In some implementations, the consistency check module further includes: The adjustment submodule is configured to adjust the allocation method of the specified task when it is determined that the first chip exists among the multiple heterogeneous chips, if it is determined that the first chip performs the specified task, and instruct the second chip to perform the specified task, and the calculation accuracy of the second chip is higher than the calculation accuracy of the first chip.

[0124] In some implementations, the precision calibration device 300 between heterogeneous chips further includes: A computational graph analysis module is configured to perform computational graph analysis on the target model to obtain a computational graph analysis result; An identification module is configured to perform node attribute analysis and key operator identification according to the calculation graph analysis result to obtain a key path analysis result; The tracking point insertion module is configured to generate a tracking point strategy according to the key path analysis result, and insert tracking points into the target model based on the tracking point strategy.

[0125] In some embodiments, the generation module 320 is configured to receive the operation results of the target model at the buried point sent by each of the heterogeneous chips; obtain the error between the operation result and the standard operation result, and write the error into the accuracy detection report.

[0126] In some implementations, the generation module 320 is further configured to mark the error when it is determined that the error exceeds a preset error, and write the marked error into the accuracy detection report.

[0127] In some implementations, the generation module 320 is further configured to perform dynamic precision compensation on the operation result when it is determined that the error exceeds a preset error.

[0128] The disclosed embodiment can improve the accuracy of target model training by calibrating the computing accuracy of multiple heterogeneous chips. Specifically, an accuracy test instruction is sent to each heterogeneous chip respectively, and the accuracy test instruction is used to instruct each heterogeneous chip to run the target model. On this basis, the operation result for the target model sent by each heterogeneous chip is received, and the operation result is compared with the standard operation result to obtain an accuracy detection report, wherein the accuracy detection report includes the accuracy error information of each heterogeneous chip. Finally, based on the accuracy detection report, the computing accuracy of multiple heterogeneous chips can be accurately calibrated. Since the accuracy detection report is obtained by letting each heterogeneous chip run the target model, the accuracy alignment of the chips can be achieved more accurately and efficiently based on the report.

[0129] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0130] Figure 8 FIG. 7 is a block diagram of an electronic device 700 according to an exemplary embodiment. Figure 8As shown, the electronic device 700 may include: a processor 701 , a memory 702 . The electronic device 700 may also include one or more of a multimedia component 703 , an input / output (I / O) interface 704 , and a communication component 705 .

[0131] The processor 701 is used to control the overall operation of the electronic device 700 to complete all or part of the steps in the above-mentioned precision calibration method between heterogeneous chips. The memory 702 is used to store various types of data to support the operation of the electronic device 700. For example, these data may include instructions for any application or method used to operate on the electronic device 700, and application-related data, such as contact data, messages sent and received, pictures, audio, video, etc. The memory 702 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (Static Random Access Memory, referred to as SRAM), electrically erasable programmable read-only memory (Electrically Erasable Programmable Read-Only Memory, referred to as EEPROM), erasable programmable read-only memory (Erasable Programmable Read-Only Memory, referred to as EPROM), programmable read-only memory (Programmable Read-Only Memory, referred to as PROM), read-only memory (Read-Only Memory, referred to as ROM), magnetic memory, flash memory, disk or optical disk. The multimedia component 703 may include a screen and an audio component. The screen may be, for example, a touch screen, and the audio component is used to output and / or input audio signals. For example, the audio component may include a microphone for receiving external audio signals. The received audio signal may be further stored in the memory 702 or sent through the communication component 705. The audio component also includes at least one speaker for outputting audio signals. The I / O interface 704 provides an interface between the processor 701 and other interface modules, and the above-mentioned other interface modules may be a keyboard, a mouse, buttons, etc. These buttons may be virtual buttons or physical buttons. The communication component 705 is used for wired or wireless communication between the electronic device 700 and other devices. Wireless communication, such as Wi-Fi, Bluetooth, Near Field Communication (NFC), 2G, 3G or 4G, or a combination of one or more of them, so the corresponding communication component 705 may include: Wi-Fi module, Bluetooth module, NFC module.

[0132] In an exemplary embodiment, the electronic device 700 can be implemented by one or more application specific integrated circuits (ASIC), digital signal processors (DSP), digital signal processing devices (DSPD), programmable logic devices (PLD), field programmable gate arrays (FPGA), controllers, microcontrollers, microprocessors or other electronic components to perform the above-mentioned precision calibration method between heterogeneous chips.

[0133] In another exemplary embodiment, a computer-readable storage medium including program instructions is also provided, and when the program instructions are executed by a processor, the steps of the above-mentioned precision calibration method between heterogeneous chips are implemented. For example, the computer-readable storage medium can be the above-mentioned memory 702 including program instructions, and the above-mentioned program instructions can be executed by the processor 701 of the electronic device 700 to complete the above-mentioned precision calibration method between heterogeneous chips.

[0134] In another exemplary embodiment, a computer program product is also provided. The computer program product includes a computer program that can be executed by a processor. When the computer program is executed by the processor, the steps of the above-mentioned accuracy calibration method between heterogeneous chips are implemented.

[0135] Fig. 9 1 is a block diagram of a server 1900 according to an exemplary embodiment. For example, the server 1900 may be provided as a server. Fig. 9 The server 1900 includes a processor 1922, which may be one or more, and a memory 1932 for storing a computer program executable by the processor 1922. The computer program stored in the memory 1932 may include one or more modules each corresponding to a set of instructions. In addition, the processor 1922 may be configured to execute the computer program to perform the above-mentioned accuracy calibration method between heterogeneous chips.

[0136] In addition, the server 1900 may further include a power supply component 1926 and a communication component 1950, wherein the power supply component 1926 may be configured to perform power management of the server 1900, and the communication component 1950 may be configured to implement communication, such as wired or wireless communication, of the server 1900. In addition, the server 1900 may further include an input / output (I / O) interface 1958. The server 1900 may operate based on an operating system stored in the memory 1932, such as Windows ServerTM, Mac OSXTM, UnixTM, LinuxTM, etc.

[0137] In another exemplary embodiment, a computer-readable storage medium including program instructions is also provided, and when the program instructions are executed by a processor, the steps of the above-mentioned precision calibration method between heterogeneous chips are implemented. For example, the computer-readable storage medium can be the above-mentioned memory 1932 including program instructions, and the above-mentioned program instructions can be executed by the processor 1922 of the server 1900 to complete the above-mentioned precision calibration method between heterogeneous chips.

[0138] Those skilled in the art may also understand that the various illustrative logical blocks and steps listed in the embodiments of the present application may be implemented by electronic hardware, computer software, or a combination of the two. Whether such functions are implemented by hardware or software depends on the specific application and the design requirements of the entire system. Those skilled in the art may use various methods to implement the functions described for each specific application, but such implementation should not be understood as exceeding the scope of protection of the embodiments of the present application.

[0139] It should be understood that, unless otherwise specifically noted, the features of some embodiments of the various present disclosures described herein may be combined with each other. As used herein, the term "and / or" includes any one of the related listed items and any combination of any two or more; similarly, "at least one of . . . " includes any one of the related listed items and any combination of any two or more.

[0140] Although terms such as "first", "second" and "third" can be used herein to describe various components, parts, regions, layers or sections, these components, parts, regions, layers or sections are not limited to these terms. On the contrary, these terms are only used to distinguish one component, component, region, layer or section from another component, component, region, layer or section. Therefore, without departing from the teachings of each example, the first component, component, region, layer or section mentioned in the examples described herein may also be referred to as the second component, component, region, layer or section. In addition, the terms "first" and "second" are only used for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features. Thus, the features defined as "first" and "second" may expressly or implicitly include at least one of the features. In the description herein, the meaning of "multiple" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0141] In addition, the word "exemplary" is used herein to indicate serving as an example, instance, or diagram. Any aspect or design described as "exemplary" in this article is not necessarily understood to be advantageous compared to other aspects or designs. On the contrary, the use of the word exemplary is intended to present concepts in a specific way. As used herein, the term "or" is intended to represent an inclusive "or" rather than an exclusive "or". That is, unless otherwise specified or clear from the context, "X applies A or B" is intended to represent any one of the natural inclusive arrangements. That is, if X applies A; X applies B; or X applies both A and B, "X applies A or B" is satisfied under any of the aforementioned examples. In addition, unless otherwise specified or clearly pointed to a singular form from the context, the articles "one" and "an" as used in this application and the appended claims are generally understood to mean "one or more".

[0142] Likewise, although the present disclosure has been shown and described with respect to one or more implementations, equivalent variations and modifications will occur to those skilled in the art after reading and understanding the specification and drawings. The present disclosure includes all such modifications and variations and is limited only by the scope of the claims. In particular, with respect to the various functions performed by the components (e.g., elements, resources, etc.) described above, unless otherwise indicated, the terms used to describe such components are intended to correspond to any component (functionally equivalent) that performs the specific functions of the described components, even if the structure is not equivalent to the disclosed structure. In addition, although specific features of the present disclosure may have been disclosed with respect to only one of several implementations, such features may be combined with one or more other features of other implementations as may be desired and beneficial to any given or specific application. In addition, with respect to "including", "having", "having", "having", or variations thereof used in a specific embodiment or claim, such terms are intended to be inclusive in a manner similar to the term "comprising".

[0143] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any modification, use or adaptation of the present disclosure, which follows the general principles of the present disclosure and includes common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present disclosure are indicated by the appended claims.

[0144] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A precision calibration method between heterogeneous chips, characterized in that: Applied to a precision calibration system, the precision calibration system includes a plurality of heterogeneous chips, and the method includes: Sending an accuracy test instruction to each of the heterogeneous chips respectively, wherein the accuracy test instruction is used to instruct each of the heterogeneous chips to run a target model; Receiving a calculation result for the target model sent by each of the heterogeneous chips, and comparing the calculation result with a standard calculation result to obtain an accuracy detection report, wherein the accuracy detection report includes accuracy error information of each of the heterogeneous chips; The calculation accuracy of the plurality of heterogeneous chips is calibrated based on the accuracy detection report.

2. The precision calibration method between heterogeneous chips according to claim 1, characterized in that: After calibrating the calculation accuracy of the plurality of heterogeneous chips based on the accuracy detection report, the method further comprises: Instructing the multiple heterogeneous chips to collaboratively train the target model; For the training of the target model, collecting the intermediate calculation results of each of the heterogeneous chips; Comparing the intermediate calculation results of the plurality of heterogeneous chips to obtain a comparison result; A consistency check is performed on the plurality of heterogeneous chips according to the comparison result.

3. The precision calibration method between heterogeneous chips according to claim 2, characterized in that: The performing consistency check on the plurality of heterogeneous chips according to the comparison result includes: Determine whether there is a first chip among the plurality of heterogeneous chips whose comparison result exceeds a preset error; If it is determined that the first chip exists, the training of the target model is terminated, and the calculation accuracy of the first chip is recalibrated.

4. The precision calibration method between heterogeneous chips according to claim 3, characterized in that: If it is determined that the first chip exists, terminating the training of the target model and recalibrating the calculation accuracy of the first chip, including: When it is determined that the first chip exists among the plurality of heterogeneous chips, if it is determined that the first chip performs a designated task, the precision correction strength of the first chip is increased.

5. The precision calibration method between heterogeneous chips according to claim 4, characterized in that: The method further comprises: When it is determined that the first chip exists among the multiple heterogeneous chips, if it is determined that the first chip performs a specified task, the allocation method of the specified task is adjusted to instruct a second chip to perform the specified task, and the calculation accuracy of the second chip is higher than that of the first chip.

6. The accuracy calibration method between heterogeneous chips according to claim 1, characterized in that: Before sending the accuracy test instruction to each of the heterogeneous chips respectively, the method includes: Performing a computational graph analysis on the target model to obtain a computational graph analysis result; Perform node attribute analysis and key operator identification according to the calculation graph analysis results to obtain key path analysis results; A tracking strategy is generated according to the critical path analysis result, and tracking points are inserted into the target model based on the tracking strategy.

7. The precision calibration method between heterogeneous chips according to claim 6, characterized in that: The receiving the operation result for the target model sent by each of the heterogeneous chips, and comparing the operation result with the standard operation result to obtain an accuracy detection report, includes: Receiving a calculation result of the target model at the embedding point sent by each of the heterogeneous chips; Obtain an error between the calculation result and the standard calculation result, and write the error into the accuracy detection report.

8. The precision calibration method between heterogeneous chips according to claim 7, characterized in that: The step of writing the error into the accuracy detection report comprises: When it is determined that the error exceeds the preset error, the error is marked, and the marked error is written into the accuracy detection report.

9. The precision calibration method between heterogeneous chips according to claim 8, characterized in that: The method further comprises: When it is determined that the error exceeds a preset error, dynamic accuracy compensation is performed on the calculation result.

10. A precision calibration device between heterogeneous chips, characterized in that: Applied to a precision calibration system, the precision calibration system includes a plurality of heterogeneous chips, and the device includes: A sending module is configured to send an accuracy test instruction to each of the heterogeneous chips respectively, wherein the accuracy test instruction is used to instruct each of the heterogeneous chips to run a target model; A generation module is configured to receive a calculation result for the target model sent by each of the heterogeneous chips, and compare the calculation result with a standard calculation result to obtain an accuracy detection report, wherein the accuracy detection report includes accuracy error information of each of the heterogeneous chips; The calibration module is configured to calibrate the calculation accuracy of the plurality of heterogeneous chips based on the accuracy detection report.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method described in any one of claims 1 to 9 are implemented.

12. An electronic device, characterized in that: include: a memory having a computer program stored thereon; A processor, configured to execute the computer program in the memory to implement the steps of the method according to any one of claims 1 to 9.

13. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 9 are implemented.

Images