Operator inspection method and device and storage medium

Abstract

The embodiment of the invention provides an operator checking method and device and a storage medium, which are used for checking out an abnormal operator in advance and reducing the downtime and restart times of an artificial intelligence chip. The method comprises the steps that before a first process sends a calculation task of a first operator to an artificial intelligence chip, a key behavior of the first operator running on the artificial intelligence chip is simulated, simulation data corresponding to the key behavior are obtained, and the first process is any process running on a main control chip; checking the first operator according to the simulation data to obtain a first checking result; if the first check result is that the first operator is abnormal, an interrupt request is initiated to the first process, and the interrupt request is used for requesting to interrupt the calculation task of sending the first operator to the artificial intelligence chip. Therefore, the abnormal operator can be checked out in advance, and the interrupt request is initiated, so that the calculation task of sending the operator to the artificial intelligence chip is interrupted, and the downtime and restart times of the artificial intelligence chip are reduced.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of artificial intelligence technology, and in particular to an operator checking method, device and storage medium. Background Art

[0002] In the actual application of artificial intelligence chips, computing tasks of multiple operators initiated by multiple users can be run on the artificial intelligence chip at the same time. If an abnormality occurs in one of the operators, it is usually not handled in the relevant technology, which will cause problems in the entire artificial intelligence chip. For example, the artificial intelligence chip may crash, which in turn affects the normal execution of computing tasks of other running operators. The computing tasks of the operators can only be re-run by restarting the artificial intelligence chip, and the artificial intelligence chip needs to be restarted many times. Summary of the invention

[0003] Embodiments of the present invention provide an operator checking method, device, and storage medium for detecting abnormal operators in advance and reducing the number of downtime and restart times of an artificial intelligence chip.

[0004] In a first aspect, the present application provides an operator checking method, which is applied to a main control chip. The method includes: before a first process sends a computing task of a first operator to an artificial intelligence chip, simulating a key behavior of the first operator when running on the artificial intelligence chip, and obtaining simulation data corresponding to the key behavior, where the first process is any process running on the main control chip; checking the first operator according to the simulation data to obtain a first inspection result; if the first inspection result is that an abnormality occurs in the first operator, initiating an interrupt request to the first process, and the interrupt request is used to request to interrupt the computing task of sending the first operator to the artificial intelligence chip.

[0005] Through the above scheme, before the main control chip sends a computing task of an operator to the artificial intelligence chip, it first simulates the key behavior of the operator when it is running on the artificial intelligence chip, and checks the operator based on the simulation data corresponding to the key behavior, so that the operator with abnormalities can be detected in advance before the first process sends the operator task to the artificial intelligence chip, and when the operator is detected to be abnormal, an interrupt request is initiated to the first process, thereby interrupting the computing task of sending the operator to the artificial intelligence chip, reducing the number of downtime and restart times of the artificial intelligence chip.

[0006] In a possible implementation, the key behavior includes at least one of the following behaviors:

[0007] Calculate target memory addresses accessed by multiple software threads corresponding to the first operator respectively;

[0008] Synchronization between software threads with data dependency corresponding to the first operator;

[0009] Different hardware resources with dependencies managed by a software thread corresponding to the first operator are synchronized.

[0010] In a possible implementation, the first operator is checked according to simulation data corresponding to the key behavior to obtain a first inspection result, including: checking the key behavior corresponding to the first operator according to the simulation data; if the inspection result corresponding to any one of the key behaviors corresponding to the first operator is failed, then determining that the first inspection result is that the first operator is abnormal; or, if the inspection result corresponding to each of the key behaviors corresponding to the first operator is passed, then determining that the first inspection result is that the first operator is normal.

[0011] In one possible implementation, the first operator is checked according to simulation data to obtain a first inspection result, including: checking the key behaviors corresponding to the first operator according to the simulation data, and checking the memory leak of the random dynamic memory corresponding to the first operator; if the inspection result corresponding to any one of the key behaviors corresponding to the first operator is failed, or the result of the inspection of the random dynamic memory leak corresponding to the first operator is failed, then it is determined that the first inspection result is that the first operator is abnormal; or, if the inspection result corresponding to each of the key behaviors corresponding to the first operator is passed, and the result of the inspection of the random dynamic memory leak corresponding to the first operator is passed, then it is determined that the first inspection result is that the first operator is normal.

[0012] In one possible implementation, a memory leak check of a random dynamic memory corresponding to a first operator includes: checking an application status of a memory space corresponding to the first operator and a release status of the applied memory space; if it is determined that the released capacity of the memory space corresponding to the first operator is consistent with the applied capacity of the memory space corresponding to the first operator, then determining that a result of the memory leak check of the random dynamic memory corresponding to the first operator is passed; or, if it is determined that the released capacity of the memory space corresponding to the first operator is inconsistent with the applied capacity of the memory space corresponding to the first operator, then determining that a result of the memory leak check of the random dynamic memory corresponding to the first operator is failed.

[0013] In one possible implementation, the key behavior includes calculating target memory addresses respectively accessed by multiple software threads corresponding to the first operator, and the simulation data includes target memory addresses respectively corresponding to the multiple software threads corresponding to the first operator; the key behavior corresponding to the first operator is checked according to the simulation data, including: if it is determined according to the simulation data that the target memory address accessed by any software thread among the multiple software threads is not within the target address range, then the result of the check on the memory access address corresponding to the first operator is determined to be failed; or, if it is determined according to the simulation data that the target memory address accessed by each software thread among the multiple software threads is within the target address range, then the result of the check on the memory access address corresponding to the first operator is determined to be passed.

[0014] In one possible implementation, the critical behavior includes synchronization between software threads with data dependencies corresponding to the first operator, and the simulation data includes the number of first synchronization instructions executed by all first threads in multiple groups of software threads with data dependencies corresponding to the first operator and the number of first wait synchronization instructions executed by all second threads in the multiple groups of software threads; the critical behavior corresponding to the first operator is checked according to the simulation data, including: if it is determined according to the simulation data that the sum of the number of first synchronization instructions executed by all first threads in the multiple groups of software threads with data dependencies corresponding to the first operator is consistent with the sum of the number of first wait synchronization instructions executed by all second threads in the multiple groups of software threads, then the result of the software thread synchronization check for the first operator is determined to be passed; or, if it is determined according to the simulation data that the sum of the number of first synchronization instructions executed by all first threads in the multiple groups of software threads with data dependencies corresponding to the first operator is inconsistent with the sum of the number of first wait synchronization instructions executed by all second threads in the multiple groups of software threads, then the result of the software thread synchronization check for the first operator is determined to be failed.

[0015] In one possible implementation, the key behavior includes synchronization of different hardware resources with dependencies managed by a software thread corresponding to the first operator, the different hardware resources with dependencies include a first hardware resource and a second hardware resource, and the simulation data includes the number of second synchronization instructions sent by a software thread to the first hardware resource and the number of second wait synchronization instructions sent by a software thread to the second hardware resource; the key behavior corresponding to the first operator is checked according to the simulation data, including: if it is determined according to the simulation data that the number of second synchronization instructions sent by a software thread to the first hardware resource and the number of second wait synchronization instructions sent to the second hardware resource are consistent, then it is determined that a result of the synchronization check of different hardware resources with dependencies managed by a software thread corresponding to the first operator is passed; or, if it is determined according to the simulation data that the number of second synchronization instructions sent by a software thread to the first hardware resource and the number of second wait synchronization instructions sent to the second hardware resource are inconsistent, then it is determined that a result of the synchronization check of different hardware resources with dependencies managed by a software thread corresponding to the first operator is failed.

[0016] In a possible implementation, the method further includes: if the first check result is that the first operator is normal, sending the computing task of the first operator to the artificial intelligence chip through the first process.

[0017] In a second aspect, the present application provides an operator checking device, comprising:

[0018] a simulation unit, configured to simulate a key behavior of the first operator when it is running on the artificial intelligence chip before the first process sends a computing task of the first operator to the artificial intelligence chip, and obtain simulation data corresponding to the key behavior, wherein the first process is any process running on the main control chip;

[0019] A checking unit, configured to check the first operator according to the simulation data to obtain a first checking result;

[0020] The processing unit is used to initiate an interrupt request to the first process if the first check result is that the first operator is abnormal, and the interrupt request is used to request the interruption of sending the computing task of the first operator to the artificial intelligence chip.

[0021] In a third aspect, the present application provides a computer device, comprising a memory, a processor chip, and a computer program stored in the memory and executable on the processor chip, wherein the processor chip implements the method in the above-mentioned first aspect or any possible implementation manner of the first aspect when executing the computer program.

[0022] In a fourth aspect, the present application provides a computer-readable storage medium, comprising computer-executable instructions. When the computer-executable instructions are executed on a computer, the computer executes the method in the above-mentioned first aspect or any possible implementation manner of the first aspect.

[0023] In a fifth aspect, the present application provides a computer program product, wherein the computer program product stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by a computer, the computer executes the method in the above-mentioned first aspect or any possible implementation manner of the first aspect.

[0024] The technical effects that can be achieved in any of the second to fifth aspects mentioned above can refer to the description of the beneficial effects in the first aspect mentioned above or any possible implementation method of the first aspect, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 A schematic diagram of a system architecture applicable to an embodiment of the present application;

[0026] Figure 2 A flowchart of an operator checking method provided by an embodiment of the present invention;

[0027] Figure 3 A schematic diagram of the structure of an operator checking device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0028] In order to make the purpose, technical solution and beneficial effects of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0029] Figure 1 This is a system architecture diagram applicable to the embodiments of the present application. The system architecture includes a main control chip 100 and an artificial intelligence chip 200. The number of artificial intelligence chips 200 can be one or more. The present application does not specifically limit the number of artificial intelligence chips. It should be understood that the system architecture may also include other modules or components, which are not specifically limited in the present application. For ease of description, the following embodiments are described by taking an artificial intelligence chip as an example.

[0030] The main control chip 100 can process non-computing tasks in artificial intelligence applications.

[0031] The artificial intelligence chip 200 can handle a large number of computing tasks in artificial intelligence applications. For example, the operation of the artificial intelligence model can be realized through the artificial intelligence chip 200. The operation of the artificial intelligence model can be implemented by the operators in the computational graph, where the computational graph is a multi-graph structure used to represent the computing tasks and data flow processes of the artificial intelligence model, and the operator refers to various operations performed on the tensors of each layer in the artificial intelligence model.

[0032] In actual applications, the main control chip 100 can send the computing tasks of the operators in the artificial intelligence model to the artificial intelligence chip 200, and the artificial intelligence chip 200 executes the computing tasks of the operators sent by the main control chip 100.

[0033] The artificial intelligence chip includes at least video memory and multiple computing cores. The video memory is used to store data and tasks sent by the main control chip and the results of the computing core after executing the task. It can be high bandwidth memory (HBM) or other types of memory.

[0034] The computing core is a computing unit that performs computing tasks in an artificial intelligence chip, and can be a programmable multiprocessor, such as a streaming processing cluster (SPC). Each programmable multiprocessor includes multiple execution units, each of which includes on-chip cache, registers, tensor cores, and thread bundles. The thread bundles include multiple threads (for example, 32 threads). The thread bundles in the artificial intelligence chip can be called by the main control chip to implement the computing tasks of the execution operator.

[0035] It should be understood that the present application does not limit the number of computing cores included in the artificial intelligence chip. The artificial intelligence chip may also include other modules or components, which are not limited in the present application.

[0036] The main control chip in the embodiment of the present application may be, for example, a central processing unit (CPU), and the artificial intelligence chip is also called an AI accelerator or computing card. The artificial intelligence chip may be, for example, a graphics processing unit (GPU) or a general-purpose computing on graphics processing units (GPGP U).

[0037] In the actual application of artificial intelligence chips, computing tasks of multiple operators initiated by multiple users can be run on the artificial intelligence chip at the same time. If an abnormality occurs in one of the operators, it is usually not handled in the relevant technology, which will cause problems in the entire artificial intelligence chip. For example, the artificial intelligence chip may crash, which in turn affects the normal execution of computing tasks of other running operators. The computing tasks of the operators can only be re-run by restarting the artificial intelligence chip, and the artificial intelligence chip needs to be restarted many times.

[0038] In view of this, the present application provides a process of an operator inspection method, the process of the method can be executed by an operator inspection device, and the operator inspection device can be a main control chip or a software program running in the main control chip, such as Figure 2 As shown, the method comprises the following steps:

[0039] Step 201, before the first process sends the computing task of the first operator to the artificial intelligence chip, simulate the key behavior of the first operator when running on the artificial intelligence chip to obtain simulation data corresponding to the key behavior.

[0040] At least one artificial intelligence application is running on the main control chip, and each artificial intelligence application corresponds to a process. Each of the multiple processes running on the main control chip can send the computing task of the operator to the artificial intelligence chip, and each process can initiate the computing task of one or more operators. The first process in the above step 201 can be any process running on the main control chip, the first process can send the computing tasks of multiple operators to the artificial intelligence chip, and the first operator can be any operator corresponding to the computing tasks of multiple operators sent by the first process. That is to say, for any process running on the main control chip, before this process sends the computing task of each operator to the artificial intelligence chip, the operator inspection device must first simulate the key behavior of each operator when running on the artificial intelligence chip, obtain the simulation data corresponding to the key behavior, and then check each operator according to the simulation data corresponding to the key behavior of each operator when running on the artificial intelligence chip.

[0041] Among them, key behaviors include but are not limited to at least one of the following:

[0042] (1) calculating target memory addresses accessed by multiple software threads corresponding to the first operator respectively;

[0043] (2) synchronization between software threads with data dependencies corresponding to the first operator;

[0044] (3) Synchronization of different hardware resources with dependencies managed by a software thread corresponding to the first operator.

[0045] The key behaviors mentioned above include different contents, and the simulation data corresponding to the key behaviors are also different.

[0046] Before the first process sends the computing task of the first operator to the artificial intelligence chip, the first process will initiate many software threads to process the computing task of the first operator. These software threads will be deployed on the artificial intelligence chip and execute the computing task of the first operator through the hardware resources on the artificial intelligence chip. Hereinafter, the multiple software threads corresponding to the first operator can be understood as software threads initiated by the first process to process the computing task of the first operator.

[0047] The following describes the case where the key behavior includes the above-mentioned behavior (1).

[0048] When the first operator runs on the artificial intelligence chip, the software thread corresponding to the first operator needs to read the data to be processed from the video memory on the artificial intelligence chip. The data to be processed may include multiple elements. For multiple software threads that need to read the data to be processed from the video memory, each of the software threads will process different elements in the video memory, and it is necessary to access the target memory address of this element in the video memory. Therefore, the main control chip can simulate the behavior of calculating the target memory addresses accessed by the multiple software threads corresponding to the first operator respectively when the first operator runs on the artificial intelligence chip, and obtain the corresponding simulation data. When the key behavior includes the above-mentioned behavior (1), the corresponding simulation data is the target memory addresses accessed by the multiple software threads corresponding to the first operator respectively.

[0049] The following describes the case where the key behavior includes the above-mentioned behavior (2).

[0050] When the first operator runs on the artificial intelligence chip, the multiple software threads corresponding to the first operator include multiple groups of software threads with data dependencies, and each group of software threads with data dependencies includes two software threads, which are called the first thread and the second thread respectively. For example, the data required for thread 2 to perform the computing task depends on the computing result of thread 1, and the data required for thread 3 to perform the computing task depends on the computing result of thread 2, then thread 1 and thread 2 are a group of software threads with data dependencies, and thread 2 and thread 3 are a group of software threads with data dependencies.

[0051] Taking the example that the data required by the second thread to perform a computing task depends on the computing result of the first thread, the first thread is used to execute a first synchronization instruction, which is used to instruct the first thread to send a synchronization signal to the second thread after completing its own computing task and obtaining the computing result, and the synchronization signal is used to indicate that the first thread has completed its own computing task and obtained the data that the second thread depends on; the second thread is used to execute a first waiting synchronization instruction, which is used to instruct the second thread to wait for the first thread to send a synchronization signal, and then start computing after receiving the synchronization signal. Therefore, the main control chip can simulate the synchronization behavior between the software threads with data dependency corresponding to the first operator when the first operator is running on the artificial intelligence chip, and can obtain the corresponding simulation data. When the key behavior includes the above behavior (2), the corresponding simulation data is the sum of the number of first synchronization instructions executed by all first threads in the multiple groups of software threads with data dependency corresponding to the first operator and the sum of the number of first waiting synchronization instructions executed by all second threads in the multiple groups of software threads.

[0052] The following describes the case where the key behavior includes the above-mentioned behavior (3).

[0053] When the first operator is running on the artificial intelligence chip, the multiple software threads corresponding to the first operator include software threads for managing different hardware resources. A software thread can manage different hardware resources with data dependencies through instructions. Taking the example that the data required for the execution process of the second hardware resource depends on the execution result of the first hardware resource, the software thread can send a second synchronization instruction to the first hardware resource, and send a second wait synchronization instruction to the second hardware resource.

[0054] For example, a software thread needs to load data from storage space and complete the calculation of the loaded data through different hardware resources in the artificial intelligence chip. The process includes: the hardware loading resource loads data from the storage space to the hardware computing resource, and sends a synchronization signal to the hardware computing resource after the data loading is completed. After the hardware computing resource receives the synchronization signal, it calculates the loaded data. The two hardware resources with data dependency involved in this process include hardware computing resources and hardware loading resources. The data required for the hardware computing resource to perform the computing task depends on the data loading result of the hardware loading resource. The hardware computing resource is, for example, the tensor core in the artificial intelligence chip, and the hardware loading resource is, for example, the loading storage controller in the artificial intelligence chip used to load data from the video memory to the tensor core.

[0055] Specifically, the software thread can send a second synchronization instruction to the hardware loading resource, and send a second waiting synchronization instruction to the hardware computing resource, wherein the second synchronization instruction is used to instruct the hardware loading resource to send a synchronization signal to the hardware computing resource after loading the data, and the synchronization signal is used to indicate that the hardware loading resource has completed data loading; the second waiting synchronization instruction is used to instruct the hardware computing resource to wait for the hardware loading resource to send a synchronization signal, and then perform calculations after receiving the synchronization signal. Therefore, the main control chip can simulate the behavior of synchronizing the hardware computing resource and the hardware loading resource managed by a software thread corresponding to the first operator when the first operator is running on the artificial intelligence chip, and can obtain the corresponding simulation data, which is the number of second synchronization instructions sent by a software thread to the hardware loading resource and the number of second waiting synchronization instructions sent to the hardware computing resource. It should be understood that here, the different hardware resources with dependencies managed by the software thread are hardware computing resources and hardware loading resources with data dependencies. The embodiment of the present application does not limit the different hardware resources with dependencies managed by the software thread, and the embodiment of the present application is also applicable to other different hardware resources with data dependencies.

[0056] In an embodiment of the present application, when the key behavior includes the above-mentioned behavior (3), its corresponding simulation data includes the number of second synchronization instructions sent by a software thread to the first hardware resource managed by itself and the number of second wait synchronization instructions sent by a software thread to the second hardware resource managed by itself.

[0057] Step 202: Check the first operator according to the simulation data to obtain a first check result.

[0058] In a possible implementation, step 202 may be implemented in the following manner: checking the key behaviors corresponding to the first operator based on simulation data; if any of the key behaviors corresponding to the first operator has a failed inspection result, determining that the first inspection result is that the first operator is abnormal; or, if each of the key behaviors corresponding to the first operator has a passed inspection result, determining that the first inspection result is that the first operator is normal.

[0059] Based on the different contents included in the key behaviors in the above step 201, there are multiple possible implementations for checking each of the key behaviors corresponding to the first operator according to the simulation data.

[0060] In implementation mode one, the key behavior includes the above-mentioned behavior (1). In step 202, the memory access address check corresponding to the first operator is implemented according to the simulation data corresponding to behavior (1), that is, the target memory addresses accessed by multiple software threads corresponding to the first operator.

[0061] Specifically, if it is determined based on the simulation data that the target memory address accessed by any software thread among multiple software threads is not within the target address range, then the result of the memory access address check corresponding to the first operator is determined to be failed; or, if it is determined based on the simulation data that the target memory address accessed by each software thread among multiple software threads is within the target address range, then the result of the memory access address check corresponding to the first operator is determined to be passed.

[0062] The target address interval can be determined by the operator checking device according to the to-be-processed data corresponding to the first operator. For example, the to-be-processed data corresponding to the first operator includes 400 elements, and the target address interval is a continuous address range starting from the starting address of the first element in the video memory among the 400 elements, and the length of this continuous address range is the size of 400 elements. For example, the starting address of the first element in the video memory is the first address in the video memory, recorded as address 0, and the 400th element occupies address 399, then the target address interval is a continuous address range from address 0 to address 399, if the target memory address accessed by a certain software thread among the multiple software threads corresponding to the first operator is address 400, and address 400 is not within the target address interval (address 0 to address 399), then the result of the memory access address check corresponding to the first operator is determined to be failed; if the target memory address accessed by each software thread among all the software threads corresponding to the first operator is within the target address interval (address 0 to address 399), then the result of the memory access address check corresponding to the first operator is determined to be passed.

[0063] In implementation mode two, the key behavior includes the above-mentioned behavior (2). In step 202, according to the simulation data corresponding to behavior (2), that is, the number of first synchronization instructions executed by all first threads in the multiple groups of software threads with data dependencies corresponding to the first operator and the number of first waiting synchronization instructions executed by all second threads in the multiple groups of software threads, synchronization check between software threads corresponding to the first operator is implemented.

[0064] Specifically, if it is determined based on simulation data that the total number of first synchronization instructions executed by all first threads in a plurality of groups of software threads with data dependencies corresponding to the first operator is consistent with the total number of first waiting synchronization instructions executed by all second threads in the plurality of groups of software threads, then the result of the synchronization check between the software threads corresponding to the first operator is determined to be passed; or, if it is determined based on simulation data that the total number of first synchronization instructions executed by all first threads in a plurality of groups of software threads with data dependencies corresponding to the first operator is inconsistent with the total number of first waiting synchronization instructions executed by all second threads in the plurality of groups of software threads, then the result of the synchronization check between the software threads corresponding to the first operator is determined to be failed.

[0065] It should be noted that the operator checking device can simulate the key behavior of the first operator when it runs on the artificial intelligence chip on the main control chip, but cannot simulate the timing of instruction execution on the artificial intelligence chip. Therefore, the sum of the number of first synchronization instructions executed by all the first threads is consistent with the sum of the number of first waiting synchronization instructions executed by all the second threads. It can be considered that the first synchronization instructions executed by all the first threads and the first waiting synchronization instructions executed by all the second threads are one-to-one corresponding. In this way, it can be ensured that the calculation task of the first operator will not cause the artificial intelligence chip to crash when it is executed on the artificial intelligence chip. Even if the first synchronization instructions executed by all the first threads are not one-to-one corresponding to the first waiting synchronization instructions executed by all the second threads, when the sum of the number of first synchronization instructions executed by all the first threads is consistent with the sum of the number of first waiting synchronization instructions executed by all the second threads, the calculation result of the first operator may be abnormal when it is executed on the artificial intelligence chip, but it will not cause the artificial intelligence chip to crash.

[0066] Implementation method three, the key behavior includes the above-mentioned behavior (3), and in step 202, based on the simulation data corresponding to behavior (3), that is, the number of second synchronization instructions sent by a software thread to the first hardware resource managed by itself and the number of second wait synchronization instructions sent to the second hardware resource managed by itself, a synchronization check of different hardware resources with dependencies managed by a software thread corresponding to the first operator is implemented.

[0067] Specifically, if it is determined according to the simulation data that the number of second synchronization instructions is consistent with the number of second waiting synchronization instructions, then the result of the synchronization check of different hardware resources with dependencies managed by a software thread corresponding to the first operator is determined to be passed; or, if it is determined according to the simulation data that the number of second synchronization instructions is inconsistent with the number of second waiting synchronization instructions, then the result of the synchronization check of different hardware resources with dependencies managed by a software thread corresponding to the first operator is determined to be failed.

[0068] It should be noted that the operator checking device can simulate the key behaviors of the first operator when it runs on the artificial intelligence chip on the main control chip, but cannot simulate the timing of instruction execution on the artificial intelligence chip. Therefore, the number of the second synchronization instructions is consistent with the number of the second waiting synchronization instructions. It can be considered that the status of different hardware resources managed by the software thread is correct each time they are synchronized. In this way, it can be guaranteed that the calculation task of the first operator will not cause the artificial intelligence chip to crash when it is executed on the artificial intelligence chip. Even if the status of different hardware resources managed by the software thread is not correct every time they are synchronized, when the number of the second synchronization instructions is consistent with the number of the second waiting synchronization instructions, the calculation result of the first operator may be abnormal when it is executed on the artificial intelligence chip, but it will not cause the artificial intelligence chip to crash.

[0069] In the fourth implementation mode, the key behavior includes a combination of any number of the above-mentioned behaviors (1) to (3), and in step 202, corresponding checks are performed according to the simulation data corresponding to the combination of any number of the above-mentioned behaviors. For example, if the key behavior includes a combination of the above-mentioned behaviors (1) and (2), then the simulation data includes: target memory addresses accessed by multiple software threads corresponding to the first operator, the total number of first synchronization instructions sent by all first threads in the multiple groups of software threads with data dependencies corresponding to the first operator, and the total number of first wait synchronization instructions received by all second threads in the multiple groups of software threads. This can implement a memory access address check corresponding to the first operator and a synchronization check between software threads of the first operator.

[0070] This application does not list all combinations of any multiple of the above-mentioned behaviors (1) to (3), and reference may be made to the combination of the above-mentioned behaviors (1) and (2).

[0071] For any combination of multiple items in the above behaviors (1) to (3), if the inspection result corresponding to any behavior in the combination is failure, the first inspection result is that the first operator is abnormal; if the inspection results corresponding to all behaviors in the combination are passed, the first inspection result is that the first operator is normal.

[0072] In another possible implementation, the above step 202 can be implemented in the following manner: checking the key behaviors corresponding to the first operator and checking the memory leak of the random dynamic memory corresponding to the first operator according to the simulation data; if the inspection result corresponding to any one of the key behaviors corresponding to the first operator is failed, or the result of the random dynamic memory leak inspection corresponding to the first operator is failed, then it is determined that the first inspection result is that the first operator is abnormal; or, if the inspection result corresponding to each of the key behaviors corresponding to the first operator is passed, and the result of the random dynamic memory leak inspection corresponding to the first operator is passed, then it is determined that the first inspection result is that the first operator is normal.

[0073] The key behavior corresponding to the first operator is checked according to the simulation data, which can be seen from the description of the above-mentioned embodiments 1 to 4, and will not be described in detail here. The following describes the random dynamic memory memory leak check corresponding to the first operator.

[0074] Before the computing task of the first operator is sent to the artificial intelligence chip for execution, it will apply for memory space on the artificial intelligence chip on the main control chip to store the data of the first operator. The data in the memory space will be accessed when the first operator is running on the artificial intelligence chip. After the computing task of the first operator is completed, the corresponding memory space will be released. The operator inspection device can obtain the memory space application status corresponding to the first operator from the main control chip. After simulating the key behavior of the first operator running on the artificial intelligence chip, the operator inspection device can also obtain the release status of the memory space corresponding to the first operator from the main control chip. Therefore, the operator inspection device can check the memory space application status corresponding to the first operator and the release status of the applied memory space, and implement memory leak inspection of the random dynamic memory corresponding to the first operator.

[0075] Specifically, if the operator checking device determines that the released capacity of the memory space corresponding to the first operator is consistent with the applied capacity of the memory space corresponding to the first operator, then the result of the memory leak check of the random dynamic memory corresponding to the first operator is determined to be passed; or, if it is determined that the released capacity of the memory space corresponding to the first operator is inconsistent with the applied capacity of the memory space corresponding to the first operator, then the result of the memory leak check of the random dynamic memory corresponding to the first operator is determined to be failed.

[0076] The following is an explanation with reference to specific examples.

[0077] For example, the key behavior includes the above-mentioned behavior (1), and the above-mentioned step 202 is implemented in the following manner: according to the simulation data corresponding to the above-mentioned behavior (1), that is, the target memory addresses accessed by multiple software threads corresponding to the first operator respectively, the memory access address corresponding to the first operator is checked; and according to the application of the memory space corresponding to the first operator and the release of the applied memory space, the random dynamic memory corresponding to the first operator is checked for memory leaks; if the result of the memory access address check corresponding to the first operator is failed, or the result of the memory leak check of the random dynamic memory corresponding to the first operator is failed, then it is determined that the first check result is that the first operator is abnormal; if the result of the memory access address check corresponding to the first operator is passed, and the result of the memory leak check of the random dynamic memory corresponding to the first operator is passed, then it is determined that the first check result is that the first operator is normal. In this example, the critical behavior includes behavior (1) as an example to illustrate that the above step 202 is implemented by checking the critical behavior corresponding to the first operator and checking the memory leak of the random dynamic memory corresponding to the first operator, wherein the critical behavior includes the above behavior (2), or includes behavior (3), or includes any combination of the above behaviors (1) to (3), which will not be repeated here.

[0078] Step 203: If the first check result is that the first operator is abnormal, an interrupt request is initiated to the first process, and the interrupt request is used to request to interrupt the computing task of sending the first operator to the artificial intelligence chip.

[0079] Among them, after receiving the interrupt request, the first process no longer sends the computing task of the first operator to the artificial intelligence chip.

[0080] In the embodiment of the present application, before the main control chip sends a computing task of an operator to the artificial intelligence chip, it first simulates the key behavior of the operator when it is running on the artificial intelligence chip, and checks the operator according to the simulation data corresponding to the key behavior. It can detect the abnormal operator in advance before sending the operator task to the artificial intelligence chip, and when the operator is detected to be abnormal, it initiates an interrupt request to the first process, and then the first process interrupts the computing task of sending the operator to the artificial intelligence chip, reducing the number of downtime and restarts of the artificial intelligence chip. Moreover, in the present application, only part of the key behavior of the first operator is simulated in the main control chip, and no time-consuming numerical calculation is performed, which is efficient and will not affect the execution efficiency of the operator.

[0081] In some other embodiments, if the first check result is that the first operator is normal, the computing task of the first operator is sent to the artificial intelligence chip through the first process. That is to say, when the main control chip side checks that the first operator is normal, the operator checking device will not send an interrupt request to the first process, and the first process normally sends the computing task of the first operator to the artificial intelligence chip.

[0082] Based on the same technical concept, the present application provides an operator inspection device, such as Figure 3 As shown, the operator checking device 300 includes:

[0083] The simulation unit 301 is used to simulate the key behavior of the first operator when it is running on the artificial intelligence chip before the first process sends the computing task of the first operator to the artificial intelligence chip, and obtain simulation data corresponding to the key behavior. The first process is any process running on the main control chip;

[0084] A checking unit 302, configured to check the first operator according to the simulation data to obtain a first checking result;

[0085] The processing unit 303 is used to initiate an interrupt request to the first process if the first check result is that the first operator is abnormal, and the interrupt request is used to request to interrupt the computing task of sending the first operator to the artificial intelligence chip.

[0086] In a possible implementation, the key behavior includes at least one of the following behaviors:

[0087] Calculate target memory addresses accessed by multiple software threads corresponding to the first operator respectively;

[0088] Synchronization between software threads with data dependency corresponding to the first operator;

[0089] Different hardware resources with dependencies managed by a software thread corresponding to the first operator are synchronized.

[0090] In one possible implementation, the inspection unit 302 is specifically used to: inspect the key behaviors corresponding to the first operator according to the simulation data; if the inspection result corresponding to any one of the key behaviors corresponding to the first operator is failed, then the first inspection result is determined to be that the first operator is abnormal; or, if the inspection result corresponding to each of the key behaviors corresponding to the first operator is passed, then the first inspection result is determined to be that the first operator is normal.

[0091] In one possible implementation, the inspection unit 302 is specifically used to: inspect the key behaviors corresponding to the first operator according to the simulation data, and inspect the memory leak of the random dynamic memory corresponding to the first operator; if the inspection result corresponding to any one of the key behaviors corresponding to the first operator is failed, or the result of the inspection of the memory leak of the random dynamic memory corresponding to the first operator is failed, then it is determined that the first inspection result is that the first operator is abnormal; or, if the inspection result corresponding to each of the key behaviors corresponding to the first operator is passed, and the result of the inspection of the memory leak of the random dynamic memory corresponding to the first operator is passed, then it is determined that the first inspection result is that the first operator is normal.

[0092] In one possible implementation, the checking unit 302 is specifically used to: check the application status of the memory space corresponding to the first operator and the release status of the applied memory space; if it is determined that the released capacity of the memory space corresponding to the first operator is consistent with the applied capacity of the memory space corresponding to the first operator, then determine that the result of the memory leak check of the random dynamic memory corresponding to the first operator is passed; or, if it is determined that the released capacity of the memory space corresponding to the first operator is inconsistent with the applied capacity of the memory space corresponding to the first operator, then determine that the result of the memory leak check of the random dynamic memory corresponding to the first operator is failed.

[0093] In one possible implementation, the key behavior includes calculating the target memory addresses respectively accessed by multiple software threads corresponding to the first operator, and the simulation data includes the target memory addresses respectively accessed by multiple software threads corresponding to the first operator; the checking unit 302 is specifically used to: if it is determined according to the simulation data that the target memory address accessed by any software thread among the multiple software threads is not within the target address range, then determine that the result of the memory access address check corresponding to the first operator is failed; or, if it is determined according to the simulation data that the target memory address accessed by each software thread among the multiple software threads is within the target address range, then determine that the result of the memory access address check corresponding to the first operator is passed.

[0094] In one possible implementation, the key behavior includes synchronization between software threads with data dependencies corresponding to the first operator, and the simulation data includes the number of first synchronization instructions executed by all first threads in multiple groups of software threads with data dependencies corresponding to the first operator and the number of first waiting synchronization instructions executed by all second threads in the multiple groups of software threads; the checking unit 302 is specifically used to: if it is determined according to the simulation data that the sum of the number of first synchronization instructions executed by all first threads in the multiple groups of software threads with data dependencies corresponding to the first operator is consistent with the sum of the number of first waiting synchronization instructions executed by all second threads in the multiple groups of software threads, then the result of the software thread synchronization check for the first operator is determined to be passed; or, if it is determined according to the simulation data that the sum of the number of first synchronization instructions executed by all first threads in the multiple groups of software threads with data dependencies corresponding to the first operator is inconsistent with the sum of the number of first waiting synchronization instructions executed by all second threads in the multiple groups of software threads, then the result of the software thread synchronization check for the first operator is determined to be failed.

[0095] In one possible implementation, the key behavior includes synchronization of different hardware resources with dependencies managed by a software thread corresponding to the first operator, the different hardware resources with dependencies include a first hardware resource and a second hardware resource, and the simulation data includes the number of second synchronization instructions sent by a software thread to the first hardware resource and the number of second wait synchronization instructions sent by a software thread to the second hardware resource; the inspection unit 302 is specifically used to: if it is determined according to the simulation data that the number of second synchronization instructions sent by a software thread to the first hardware resource and the number of second wait synchronization instructions sent to the second hardware resource are consistent, then determine that the result of the synchronization check of different hardware resources with dependencies managed by a software thread corresponding to the first operator is passed; or, if it is determined according to the simulation data that the number of second synchronization instructions sent by a software thread to the first hardware resource and the number of second wait synchronization instructions sent to the second hardware resource are inconsistent, then determine that the result of the synchronization check of different hardware resources with dependencies managed by a software thread corresponding to the first operator is failed.

[0096] In a possible implementation, the processing unit 303 is further used to: if the check result shows that the first operator is normal, send the computing task of the first operator to the artificial intelligence chip through the first process.

[0097] Based on the same inventive concept, the present application provides a computer device, which includes a memory, a processor chip, and a computer program stored in the memory and executable on the processor chip. When the processor chip executes the computer program, the method steps in any of the above method embodiments are implemented.

[0098] Based on the same technical concept, an embodiment of the present application provides a computer-readable storage medium, including computer-executable instructions. When the computer-executable instructions are executed on a computer, the computer executes the method steps in any of the above method embodiments.

[0099] Based on the same technical concept, an embodiment of the present application provides a computer program product, which stores a computer program. The computer program includes program instructions. When the program instructions are executed by a computer, the computer executes the method steps in any of the above method embodiments.

[0100] Those skilled in the art should understand that the embodiments of the present application can be provided as methods, devices (equipment), systems, chips, computer-readable storage media, or computer program products. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware, which are collectively referred to as "modules" or "systems" herein.

[0101] The present application is described with reference to at least one of the following diagrams of the method, apparatus (device) or system of the present application: a flowchart, a block diagram. It should be understood that at least one of the following can be implemented by computer program instructions: each process in the flowchart, each box in the block diagram, and a combination of a process in the flowchart and a box in the block diagram.

[0102] These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device produce instructions for implementing at least one of the following: Figure 1 Devices or blocks that specify functions in a process or multiple processes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0103] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, the instruction device implementing at least one of the following: Figure 1 A function or a block specified in a process or multiple processes Figure 1 A function specified in one or more boxes.

[0104] These computer program instructions may also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, whereby the instructions executed on the computer or other programmable device provide steps for implementing at least one of the following: steps for specifying a function in one or more processes in a flowchart, steps for specifying a function in one or more boxes in a block diagram.

[0105] Although the present invention has been described in conjunction with specific features and embodiments thereof, it is apparent that various modifications and combinations may be made thereto without departing from the spirit and scope of the present invention. Accordingly, this specification and the accompanying drawings are merely exemplary illustrations of the present invention as defined by the appended claims and are deemed to cover any and all modifications, variations, combinations or equivalents within the scope of the present invention. Obviously, those skilled in the art may make various modifications and variations to the present invention without departing from the spirit and scope of the present invention. Thus, the present invention is intended to include such modifications and variations if they fall within the scope of the claims of the present invention and their equivalents.

Claims

1. An operator checking method, characterized in that: Applied to the main control chip, the method includes: Before the first process sends the computing task of the first operator to the artificial intelligence chip, a key behavior of the first operator when running on the artificial intelligence chip is simulated to obtain simulation data corresponding to the key behavior, wherein the first process is any process running on the main control chip; Checking the first operator according to the simulation data to obtain a first checking result; If the first inspection result is that the first operator is abnormal, an interrupt request is initiated to the first process, and the interrupt request is used to request the interruption of sending the computing task of the first operator to the artificial intelligence chip.

2. The method according to claim 1, characterized in that The key behaviors include at least one of the following: Calculating target memory addresses accessed by multiple software threads corresponding to the first operator respectively; synchronization between software threads having data dependency relationship corresponding to the first operator; Different hardware resources with dependencies managed by a software thread corresponding to the first operator are synchronized.

3. The method according to claim 2, characterized in that The step of checking the first operator according to the simulation data to obtain a first checking result includes: Checking the key behavior corresponding to the first operator according to the simulation data; If the inspection result corresponding to any one of the key behaviors corresponding to the first operator is failed, it is determined that the first inspection result is that the first operator is abnormal; or, If the inspection result corresponding to each of the key behaviors corresponding to the first operator is passed, it is determined that the first inspection result is that the first operator is normal.

4. The method according to claim 2, characterized in that The step of checking the first operator according to the simulation data to obtain a first checking result includes: Checking the key behavior corresponding to the first operator according to the simulation data, and checking memory leaks of the random dynamic memory corresponding to the first operator; If the check result corresponding to any one of the key behaviors corresponding to the first operator fails, or the result of the random dynamic memory memory leak check corresponding to the first operator fails, it is determined that the first check result is that the first operator is abnormal; or, If the inspection result corresponding to each of the key behaviors corresponding to the first operator is passed, and the result of the random dynamic memory memory leak inspection corresponding to the first operator is passed, it is determined that the first inspection result is that the first operator is normal.

5. The method according to claim 4, characterized in that The random dynamic memory memory leak check corresponding to the first operator includes: Checking the application status of the memory space corresponding to the first operator and the release status of the applied memory space; If it is determined that the released capacity of the memory space corresponding to the first operator is consistent with the applied capacity of the memory space corresponding to the first operator, then it is determined that the result of the memory leak check of the random dynamic memory corresponding to the first operator is passed; or, If it is determined that the released capacity of the memory space corresponding to the first operator is inconsistent with the applied capacity of the memory space corresponding to the first operator, it is determined that the result of the memory leak check of the random dynamic memory corresponding to the first operator is failed.

6. The method according to any one of claims 3 to 5, characterized in that: The key behavior includes calculating target memory addresses respectively accessed by multiple software threads corresponding to the first operator, and the simulation data includes target memory addresses respectively accessed by multiple software threads corresponding to the first operator; The checking the key behavior corresponding to the first operator according to the simulation data includes: If it is determined according to the simulation data that the target memory address accessed by any software thread among the multiple software threads is not within the target address interval, then determining that the result of checking the memory access address corresponding to the first operator is failed; or, If it is determined according to the simulation data that the target memory address accessed by each software thread in the multiple software threads is within the target address range, then it is determined that the result of checking the memory access address corresponding to the first operator is passed.

7. The method according to any one of claims 3 to 5, characterized in that: The key behavior includes synchronization between software threads with data dependencies corresponding to the first operator, and the simulation data includes the number of first synchronization instructions executed by all first threads in a plurality of groups of software threads with data dependencies corresponding to the first operator and the number of first wait synchronization instructions executed by all second threads in the plurality of groups of software threads; The checking the key behavior corresponding to the first operator according to the simulation data includes: If it is determined according to the simulation data that the sum of the number of first synchronization instructions executed by all first threads in the multiple groups of software threads with data dependency corresponding to the first operator is consistent with the sum of the number of first waiting synchronization instructions executed by all second threads in the multiple groups of software threads, then it is determined that the result of the synchronization check between the software threads corresponding to the first operator is passed; or, If it is determined based on the simulation data that the total number of first synchronization instructions executed by all first threads in a plurality of groups of software threads with data dependencies corresponding to the first operator is inconsistent with the total number of first wait synchronization instructions executed by all second threads in the plurality of groups of software threads, then it is determined that the result of the synchronization check between software threads corresponding to the first operator is failed.

8. The method according to any one of claims 3 to 5, characterized in that: The key behavior includes synchronization of different hardware resources with dependencies managed by a software thread corresponding to the first operator, the different hardware resources with dependencies include a first hardware resource and a second hardware resource, the simulation data includes the number of second synchronization instructions sent by the software thread to the first hardware resource and the number of second wait synchronization instructions sent to the second hardware resource; and checking the key behavior corresponding to the first operator according to the simulation data includes: If it is determined according to the simulation data that the number of second synchronization instructions sent by the one software thread to the first hardware resource is consistent with the number of second wait synchronization instructions sent to the second hardware resource, then it is determined that the result of the synchronization check of different hardware resources with dependencies managed by a software thread corresponding to the first operator is passed; or, If it is determined based on the simulation data that the number of second synchronization instructions sent by the software thread to the first hardware resource and the number of second wait synchronization instructions sent to the second hardware resource are inconsistent, then it is determined that the result of the synchronization check of different hardware resources with dependencies managed by a software thread corresponding to the first operator is failed.

9. The method according to claim 1, characterized in that The method further comprises: If the first inspection result is that the first operator is normal, the computing task of the first operator is sent to the artificial intelligence chip through the first process.

10. A computer device comprising a memory, a processor chip, and a computer program stored in the memory and executable on the processor chip, characterized in that: When the processor chip executes the computer program, the steps of the method according to any one of claims 1 to 9 are implemented.

11. A computer-readable storage medium, characterized in that: The method comprises computer executable instructions, which, when executed on a computer, cause the computer to execute the steps of the method according to any one of claims 1 to 9.

12. A computer program product, characterized in that The computer program product stores a computer program, wherein the computer program includes program instructions. When the program instructions are executed by a computer, the computer is caused to execute the steps of the method according to any one of claims 1 to 9.

Images