AI chip fault software adaptive recovery system and method

Abstract

The invention belongs to the technical field of chip fault processing, and particularly relates to an AI chip fault software self-adaptive recovery system and method, performance parameters and task processing data of each AI chip are collected in real time, whether an AI chip fault exists or not is judged, and if yes, the category of data currently processed by the current fault AI chip and the category of an AI task are judged; detecting performance parameters of other available AI chips and corresponding processed data and task categories; based on the residual data of the current fault AI chip and the task category, establishing the priority of the current available AI chip; allocating the available AI chips according to the priority; according to the method, the operation performance parameters and the task processing data of the AI chip with the highest priority are monitored, whether the AI chip normally operates or whether the performance parameters for processing the new task meet specified requirements is judged, the technical problems of task rescheduling and task rescheduling to the new node are solved, and it is ensured that the system can work normally.

Description

Technical Field

[0001] The present invention belongs to the technical field of chip fault processing, and in particular, relates to an AI chip fault software adaptive recovery system and method. Background Art

[0002] AI chips are modules specifically designed to handle large amounts of computing tasks in artificial intelligence applications, also known as AI accelerators or computing cards. They are mainly divided into GPU, FPGA, ASIC and other types, among which the application-specific integrated circuit ASIC is the essence of AI chips and needs to be specially designed for AI algorithms. The emergence of AI chips has changed the way computers process data, accelerated the development of the field of artificial intelligence, and applied it to fields such as face recognition, natural language processing, machine learning and data analysis.

[0003] AI chips are now used in almost all walks of life. If an AI chip fails, it is likely to cause serious consequences. For example, AI chips are used in current autonomous driving. If the AI ​​chip related to autonomous driving fails, it will cause autonomous driving failure, which is very likely to cause serious safety accidents. Therefore, when using AI chips, it is also necessary to be able to automatically monitor or automatically recover the AI ​​chip failure when the AI ​​chip fails, or even when the AI ​​chip fails, the corresponding software can achieve automatic recovery or other corresponding solutions, so that even if the AI ​​chip fails, the entire system can work normally.

[0004] The existing technology for handling AI chip failures mainly includes task rescheduling, task suspension waiting for fault recovery, and task rescheduling to a new node. Among them, task rescheduling attempts to run the program multiple times, which is only applicable to retrying failed tasks when there is no hardware failure in the AI ​​chip. Once an AI chip fails to run, multiple rescheduling cannot solve the problem. Task suspension waiting for fault recovery, this method is to suspend program execution after discovering the fault until the hardware fault is restored; then rerun the AI ​​task. This method also has the problem that once an AI chip fails and is damaged and cannot be repaired, the program cannot be restored.

[0005] To reschedule a task to a new node, you need to prepare a backup computing node. If there is no backup node, the task cannot continue to operate. Currently, the industry calls "task rescheduling to a new node" breakpoint continuation training, which is also the most mature solution for fault rescheduling. Volcano's container scheduling system does this. This method requires a backup server node as a recovery for the failed machine, and the resource configuration of the backup server needs to be consistent with the resource configuration of the current node, that is, the number of AI chip resources and the basic environment are consistent; when a faulty chip appears in the computing node, the Volcano scheduler will reschedule the task to the new backup node to continue the operation of the last backup point. The disadvantage of this method is that it wastes resources when the backup server is idle, and if there is no backup server, the task cannot be restored on the new node.

[0006] Therefore, how to overcome the technical defects of task rescheduling, task suspension waiting for fault recovery, and task rescheduling to new nodes when an AI chip fails to ensure that the system can work normally is a technical problem that needs to be solved urgently. Summary of the invention

[0007] The purpose of the present invention is to provide an AI chip fault software adaptive recovery system and method to overcome the technical defects of task rescheduling, task suspension waiting for fault recovery, and task rescheduling to a new node, so as to ensure that the system can work normally.

[0008] In a first aspect, a method for adaptively recovering software from an AI chip fault is provided, comprising the following steps:

[0009] S1: collect the performance parameters and task processing data of each AI chip in real time to determine whether there is an AI chip failure. If so, proceed to step S2. If not, continue monitoring the AI ​​chip.

[0010] S2: Determine the data category and AI task category currently being processed by the faulty AI chip;

[0011] S3: Detect the performance parameters of other available AI chips and the corresponding processed data and task categories;

[0012] S4: Establishing the priority of the currently available AI chips based on the matching degree between the remaining data and task categories of the currently faulty AI chip and the performance parameters of the available AI chips and the corresponding processed data and task categories;

[0013] S5: Distribute the remaining tasks and processing data of the currently faulty AI chip to the AI ​​chip with the highest priority for processing;

[0014] S6: Monitor the operating performance parameters and task processing data of the AI ​​chip with the highest priority, and determine whether the AI ​​chip is operating normally or whether the performance parameters for processing new tasks meet specified requirements. If the AI ​​chip is not operating normally or the performance parameters for processing new tasks do not meet specified requirements, execute step S7; if the AI ​​chip is operating normally and the performance parameters for processing new tasks meet specified requirements, continue to process new tasks and data through the AI ​​chip;

[0015] S7: Distribute the remaining tasks and processed data to the AI ​​chip of the next priority level, and determine again whether the AI ​​chip of the next priority level is operating normally or whether the performance parameters for processing new tasks meet the specified requirements according to the requirements in step S6. If the AI ​​chip is not operating normally or the performance parameters for processing new tasks do not meet the specified requirements, distribute the remaining tasks and processed data to the AI ​​chip of the next priority level until the AI ​​chip is operating normally and the performance parameters for processing new tasks meet the specified requirements.

[0016] Preferably, the performance parameters of the AI ​​chip in step S1 include processing speed, parallel computing capability, storage capacity, bandwidth and resource utilization.

[0017] Preferably, the data categories processed in step S2 include text data, voice data, image data and video data.

[0018] Preferably, in step S4, the matching degree of the available AI chips is scored by setting a matching degree scoring mechanism, and the specific process is as follows:

[0019] S41: performing a performance score on the available AI chip according to the measured performance parameters of the available AI chip;

[0020] S42: establishing a consistency scoring mechanism with available AI chips according to the remaining data and task category of the faulty AI chip, and performing a consistency scoring on each available AI chip based on the consistency scoring mechanism;

[0021] S44: Matching score is performed based on performance score and consistency score;

[0022] S45: Set a priority for each available AI chip according to the matching score value, and add a priority tag to each available chip.

[0023] Preferably, the performance scoring formula of step S41 is as follows:

[0024] P=W1*X1+W2*X2+W3*X3+···+Wn*Xn;

[0025] Among them, P is the total performance score, W1 is the weight of the first performance parameter, X1 is the score of the first performance parameter, W2 is the weight of the second performance parameter, X2 is the score of the second performance parameter, W3 is the weight of the third performance parameter, X3 is the score of the third performance parameter, W1 is the weight of the Nth performance parameter, Xn is the score of the nth performance parameter, and the performance parameter scores of X1-Xn are obtained by multiplying the ratio of the measured performance parameters to the theoretical corresponding performance parameters by the full score of 10.

[0026] Preferably, the consistency scoring mechanism of step S42 is as follows:

[0027] S421: Scoring the consistency between the remaining data category of the faulty AI chip and the processed data category of the available AI chip to obtain a score value C1;

[0028] S422: Scoring the consistency between the remaining tasks of the faulty AI chip and the processing task categories of the available AI chip to obtain a score value C2;

[0029] S423: Calculate the total consistency score C=C1+C2.

[0030] Preferably, the matching score formula in step S44 is:

[0031] M=P+C;

[0032] Where M is the matching score of a certain available AI chip, P is the total performance score, and the total consistency score.

[0033] In a second aspect, an AI chip fault software adaptive recovery system is provided, which is used to implement any one of the AI ​​chip fault software adaptive recovery methods described above, including an AI chip monitoring module, a performance scoring module, a consistency scoring module, a matching scoring module, a priority marking module and a software adaptive scheduling module;

[0034] The AI ​​chip monitoring module is used to monitor the status of all running AI chips in real time to detect whether any AI chip has abnormalities;

[0035] The performance scoring module is used to score the performance of available AI chips;

[0036] The consistency scoring module is used to score the consistency of the remaining data and task categories of the available AI chip and the faulty AI chip;

[0037] The matching scoring module is used to score the matching degree of the remaining data and task categories of the available AI chip and the faulty AI chip according to the performance score and consistency score;

[0038] The priority marking module is used to mark the priority of the available AI chips according to the matching score value;

[0039] The software adaptive scheduling module is used to allocate the remaining data and tasks of the failed AI chip to the corresponding available AI chips.

[0040] The beneficial effects of the present invention include:

[0041] The present invention provides an AI chip fault software adaptive recovery system and method, which collects the performance parameters and task processing data of each AI chip in real time, determines whether there is an AI chip fault, and if so, determines the data category and AI task category currently being processed by the currently faulty AI chip; detects the performance parameters of the remaining available AI chips and the corresponding processed data and task category; establishes the priority of the currently available AI chip based on the remaining data and task category of the currently faulty AI chip; allocates the available AI chips according to the priority; monitors the operating performance parameters and task processing data of the AI ​​chip with the highest priority, determines whether the AI ​​chip is operating normally or whether the performance parameters for the new task processing meet the specified requirements, overcomes the problem that multiple reschedulings cannot solve the problem once a certain AI chip cannot operate, and ensures that the system can operate normally. At the same time, it can avoid the waste of resources caused by setting up a backup server. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 The figure is a flow chart of the method for adaptively recovering software from AI chip failure of the present invention.

[0043] Figure 2 The figure is a schematic diagram of the process of scoring the matching degree of available AI chips according to the present invention.

[0044] Figure 3 Schematic diagram of the architecture of the AI ​​chip fault software adaptive recovery system of the present invention. DETAILED DESCRIPTION

[0045] The following is combined with Figure 1 The present invention is further described in detail:

[0046] See also Figure 1 , an AI chip fault software adaptive recovery method, comprising the following steps:

[0047] S1: collect the performance parameters and task processing data of each AI chip in real time to determine whether there is an AI chip failure. If so, proceed to step S2. If not, continue monitoring the AI ​​chip.

[0048] S2: Determine the data category and AI task category currently being processed by the faulty AI chip;

[0049] S3: Detect the performance parameters of other available AI chips and the corresponding processed data and task categories;

[0050] S4: Establishing the priority of the currently available AI chips based on the matching degree between the remaining data and task categories of the currently faulty AI chip and the performance parameters of the available AI chips and the corresponding processed data and task categories;

[0051] S5: Distribute the remaining tasks and processing data of the currently faulty AI chip to the AI ​​chip with the highest priority for processing;

[0052] S6: Monitor the operating performance parameters and task processing data of the AI ​​chip with the highest priority, and determine whether the AI ​​chip is operating normally or whether the performance parameters for processing new tasks meet specified requirements. If the AI ​​chip is not operating normally or the performance parameters for processing new tasks do not meet specified requirements, execute step S7; if the AI ​​chip is operating normally and the performance parameters for processing new tasks meet specified requirements, continue to process new tasks and data through the AI ​​chip;

[0053] S7: Distribute the remaining tasks and processed data to the AI ​​chip of the next priority level, and determine again whether the AI ​​chip of the next priority level is operating normally or whether the performance parameters for processing new tasks meet the specified requirements according to the requirements in step S6. If the AI ​​chip is not operating normally or the performance parameters for processing new tasks do not meet the specified requirements, distribute the remaining tasks and processed data to the AI ​​chip of the next priority level until the AI ​​chip is operating normally and the performance parameters for processing new tasks meet the specified requirements.

[0054] During the operation of the server, each AI chip needs to be monitored in real time so that when an AI chip is abnormal, the faulty AI chip can be processed in time to ensure that all tasks of the system can be processed normally. Therefore, when monitoring all running AI chips, if an AI chip is found to be abnormal or faulty, the data category and AI task category currently being processed by the faulty AI chip are determined to provide data support for the subsequent selection of available AI chips.

[0055] Then, the performance parameters of the remaining available AI chips and the corresponding processed data and task categories are detected, and the performance of the available AI chips is scored through the performance scoring module and the consistency scoring module, and the consistency of the remaining data and task categories of the faulty AI chip with the available AI chips is scored. Finally, a matching scoring module is provided to score the matching degree of the available AI chips based on the performance score and consistency score of the available AI chips.

[0056] Then, the priority order of available AI chips is established through the matching score, and the priority marking module is used to mark the priority of available AI chips according to the matching score value. Finally, the software adaptive scheduling module is used to schedule the available AI chips according to the priority order to ensure that the entire system can process all tasks normally.

[0057] In the above scheme, the performance parameters of the AI ​​chip in step S1 include processing speed, parallel computing capability, storage capacity, bandwidth and resource utilization. The data categories processed in step S2 include text data, voice data, image data and video data.

[0058] See also Figure 2 In step S4, the matching degree of the available AI chips is scored by setting a matching degree scoring mechanism. The specific process is as follows:

[0059] S41: performing a performance score on the available AI chip according to the measured performance parameters of the available AI chip;

[0060] S42: establishing a consistency scoring mechanism with available AI chips according to the remaining data and task category of the faulty AI chip, and performing a consistency scoring on each available AI chip based on the consistency scoring mechanism;

[0061] S44: Matching score is performed based on performance score and consistency score;

[0062] S45: Set a priority for each available AI chip according to the matching score value, and add a priority tag to each available chip.

[0063] The performance scoring formula in step S41 is as follows:

[0064] P=W1*X1+W2*X2+W3*X3+···+Wn*Xn;

[0065] Among them, P is the total performance score, W1 is the weight of the first performance parameter, X1 is the score of the first performance parameter, W2 is the weight of the second performance parameter, X2 is the score of the second performance parameter, W3 is the weight of the third performance parameter, X3 is the score of the third performance parameter, W1 is the weight of the Nth performance parameter, Xn is the score of the nth performance parameter, and the performance parameter scores of X1-Xn are obtained by multiplying the ratio of the measured performance parameters to the theoretical corresponding performance parameters by the full score of 10.

[0066] The consistency scoring mechanism of step S42 is as follows:

[0067] S421: Scoring the consistency between the remaining data category of the faulty AI chip and the processed data category of the available AI chip to obtain a score value C1;

[0068] S422: Scoring the consistency between the remaining tasks of the faulty AI chip and the processing task categories of the available AI chip to obtain a score value C2;

[0069] S423: Calculate the total consistency score C=C1+C2.

[0070] The matching score formula in step S44 is:

[0071] M=P+C;

[0072] Where M is the matching score of a certain available AI chip, P is the total performance score, and the total consistency score.

[0073] By scoring the performance of the available AI chips as described above, the consistency of the remaining data and task categories of the available AI chips and the faulty AI chips is scored, and the performance score and consistency score are used to score the matching degree of the remaining data and task categories of the available AI chips and the faulty AI chips. Then, the priority between all available AI chips is determined and set, and the matching degree of each available AI chip in processing the remaining data and tasks of the faulty AI chip is quantified, ensuring that when an AI chip fails, the most suitable available AI chip can be selected based on the remaining data tasks and the various performance of the available AI chips. Avoid the phenomenon of mismatch between the processing of the remaining data and tasks and the selected AI chip, and ensure that even if an AI chip fails, the most suitable AI chip can be selected from the remaining available AI chips, thereby ensuring that the remaining data and tasks of the faulty AI chip can be scheduled to the adapted AI chip for processing.

[0074] An AI chip fault software adaptive recovery system, used to implement any one of the AI ​​chip fault software adaptive recovery methods described above, see Figure 3, including an AI chip monitoring module, a performance scoring module, a consistency scoring module, a matching scoring module, a priority marking module and a software adaptive scheduling module. The AI ​​chip monitoring module is used to monitor the status of all running AI chips in real time to monitor whether there are any AI chips with abnormalities. The performance scoring module is used to score the performance of the available AI chips; the consistency scoring module is used to score the consistency of the remaining data and task categories of the available AI chips and the faulty AI chips. The matching scoring module is used to score the matching degree of the remaining data and task categories of the available AI chips and the faulty AI chips according to the performance score and the consistency score; the priority marking module is used to mark the priority of the available AI chips according to the size of the matching score value. The software adaptive scheduling module is used to realize the allocation of the remaining data and tasks of the faulty AI chips to the corresponding available AI chips.

[0075] To summarize, the performance parameters and task processing data of each AI chip are collected in real time to determine whether there is an AI chip failure. If so, determine the data category and AI task category currently being processed by the currently faulty AI chip; detect the performance parameters of the remaining available AI chips and the corresponding processed data and task categories; establish the priority of the currently available AI chip based on the remaining data and task category of the currently faulty AI chip; allocate the available AI chips according to the priority; monitor the operating performance parameters and task processing data of the AI ​​chip with the highest priority to determine whether the AI ​​chip is operating normally or whether the performance parameters for the new task processing meet the specified requirements. This overcomes the problem of task rescheduling and task rescheduling to a new node. Once an AI chip fails to operate, multiple reschedulings cannot solve the problem, ensuring that the system can work normally and avoiding the waste of resources caused by setting up a backup server.

[0076] The above-mentioned embodiments only express the specific implementation method of the present application, and the description thereof is relatively specific and detailed, but it cannot be understood as limiting the protection scope of the present application. It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the technical solution concept of the present application, and these all belong to the protection scope of the present application.

Claims

1. A method for adaptive recovery of AI chip fault software, characterized in that: The following steps are involved: S1: collect the performance parameters and task processing data of each AI chip in real time to determine whether there is an AI chip failure. If so, proceed to step S2. If not, continue monitoring the AI ​​chip. S2: Determine the data category and AI task category currently being processed by the faulty AI chip; S3: Detect the performance parameters of other available AI chips and the corresponding processed data and task categories; S4: Establishing the priority of the currently available AI chips based on the matching degree between the remaining data and task categories of the currently faulty AI chip and the performance parameters of the available AI chips and the corresponding processed data and task categories; S5: Distribute the remaining tasks and processing data of the currently faulty AI chip to the AI ​​chip with the highest priority for processing; S6: Monitor the operating performance parameters and task processing data of the AI ​​chip with the highest priority, and determine whether the AI ​​chip is operating normally or whether the performance parameters for processing new tasks meet specified requirements. If the AI ​​chip is not operating normally or the performance parameters for processing new tasks do not meet specified requirements, execute step S7; if the AI ​​chip is operating normally and the performance parameters for processing new tasks meet specified requirements, continue to process new tasks and data through the AI ​​chip; S7: Distribute the remaining tasks and processed data to the AI ​​chip of the next priority level, and determine again whether the AI ​​chip of the next priority level is operating normally or whether the performance parameters for processing new tasks meet the specified requirements according to the requirements in step S6. If the AI ​​chip is not operating normally or the performance parameters for processing new tasks do not meet the specified requirements, distribute the remaining tasks and processed data to the AI ​​chip of the next priority level until the AI ​​chip is operating normally and the performance parameters for processing new tasks meet the specified requirements.

2. According to claim 1, a method for adaptive recovery of AI chip fault software, characterized in that: The performance parameters of the AI ​​chip in step S1 include processing speed, parallel computing capability, storage capacity, bandwidth, and resource utilization.

3. The method for adaptively recovering software from AI chip failure according to claim 1, characterized in that: The data categories processed in step S2 include text data, voice data, image data and video data.

4. The method for adaptively recovering software from AI chip failure according to claim 1, characterized in that: In step S4, the matching degree of the available AI chips is scored by setting a matching degree scoring mechanism. The specific process is as follows: S41: performing a performance score on the available AI chip according to the measured performance parameters of the available AI chip; S42: establishing a consistency scoring mechanism with available AI chips according to the remaining data and task category of the faulty AI chip, and performing a consistency scoring on each available AI chip based on the consistency scoring mechanism; S44: Matching score is performed based on performance score and consistency score; S45: Set a priority for each available AI chip according to the matching score value, and add a priority tag to each available chip.

5. The method for adaptively recovering software from AI chip failure according to claim 4, characterized in that: The performance scoring formula of step S41 is as follows: P=W1*X1+W2*X2+W3*X3+···+Wn*Xn; Among them, P is the total performance score, W1 is the weight of the first performance parameter, X1 is the score of the first performance parameter, W2 is the weight of the second performance parameter, X2 is the score of the second performance parameter, W3 is the weight of the third performance parameter, X3 is the score of the third performance parameter, W1 is the weight of the Nth performance parameter, Xn is the score of the nth performance parameter, and the performance parameter scores of X1-Xn are obtained by multiplying the ratio of the measured performance parameters to the theoretical corresponding performance parameters by the full score of 10.

6. The method for adaptively recovering software from AI chip failure according to claim 4, characterized in that: The consistency scoring mechanism of step S42 is as follows: S421: Scoring the consistency between the remaining data category of the faulty AI chip and the processed data category of the available AI chip to obtain a score value C1; S422: Scoring the consistency between the remaining tasks of the faulty AI chip and the processing task categories of the available AI chip to obtain a score value C2; S423: Calculate the total consistency score C=C1+C2.

7. The method for adaptively recovering software from AI chip failure according to claim 4, characterized in that: The matching score formula in step S44 is: M=P+C; Where M is the matching score of a certain available AI chip, P is the total performance score, and the total consistency score.

8. An AI chip fault software adaptive recovery system, used to implement an AI chip fault software adaptive recovery method according to any one of claims 1 to 7, characterized in that: It includes AI chip monitoring module, performance scoring module, consistency scoring module, matching scoring module, priority marking module and software adaptive scheduling module; The AI ​​chip monitoring module is used to monitor the status of all running AI chips in real time to detect whether any AI chip has abnormalities; The performance scoring module is used to score the performance of available AI chips; The consistency scoring module is used to score the consistency of the remaining data and task categories of the available AI chip and the faulty AI chip; The matching scoring module is used to score the matching degree of the remaining data and task categories of the available AI chip and the faulty AI chip according to the performance score and consistency score; The priority marking module is used to mark the priority of the available AI chips according to the matching score value; The software adaptive scheduling module is used to allocate the remaining data and tasks of the failed AI chip to the corresponding available AI chips.

Images