A fault-tolerant rescheduling method and device for a distributed training scene

Abstract

The present application relates to the technical field of rescheduling, and provides a fault-tolerant rescheduling method and device for a distributed training scene. The method comprises the following steps: presetting a training quantity and scheduling training program instances for distributed training; listening to kernel logs of resource devices and training logs of the program instances; when hardware device information with faults and errors appears in the kernel logs, adding error markers to the hardware devices with faults and errors; checking the training logs of the program instances corresponding to the hardware devices with faults and errors, and adding error markers to the program instances with errors in the training logs; placing the hardware devices with error markers and the program instances with error markers outside the scheduling range; supplementing pre-trained program instances meeting the training quantity to the scheduling range; and scheduling the program instances in the supplemented scheduling range to the resource devices in the scheduling range for distributed training. The present application can improve the efficiency and success rate of rescheduling.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of rescheduling, in particular to a fault-tolerant rescheduling method and device for a distributed training scenario. BACKGROUND

[0002] In a model training scenario, especially in a large model training scenario, the probability of hardware device failure is high, and automatic recovery training is required after hardware failure. Fault-tolerant rescheduling is a method for automatic recovery training. Fault-tolerant rescheduling relies on the return value of a training instance. One or more training program instances in a group of distributed training program instances exit with a non-zero error code. The scheduler reschedules the training program instance that exits with a non-zero error code. New hardware resources are allocated during rescheduling, and distributed training resumes training.

[0003] However, in the fault-tolerant rescheduling method, the training program exits with a non-zero error code, which is only a specification and not a hard constraint. If the training program does not return a non-zero error according to the specification, the platform cannot trigger rescheduling, the fault-tolerant function is missing, and the fault-tolerant function is greatly dependent on the specification of the training program code. If a hard constraint is imposed on the training program, it will be intrusive. In addition, the training program instance may exit due to hardware or software problems, and different error codes need to be returned for different types of problems, which makes the fault-tolerant rescheduling function greatly dependent on the accuracy of the training program error classification. In addition, even if the training program instance returns an error code, the scheduler process reschedules, and the hardware failure device may still be within the scheduling range, and the training program instance has a probability of being scheduled back to the hardware failure node. SUMMARY

[0004] The present application aims to at least solve one of the technical problems in the related art. To this end, the present application provides a fault-tolerant rescheduling method and device for a distributed training scenario.

[0005] The present application provides a fault-tolerant rescheduling method for a distributed training scenario, comprising:

[0006] S1: presetting the number of training and scheduling training program instances, and scheduling the program instances to resource devices for distributed training;

[0007] S2: listening to the kernel log of the resource device performing distributed training and the training log of the program instance;

[0008] S3: when the hardware device information with a fault error appears in the kernel log, adding an error mark to the hardware device with a fault error; checking the training log of the program instance corresponding to the hardware device with a fault error, and adding an error mark to the program instance with an error in the training log;

[0009] S4: set a scheduling range, and place the hardware device with the error label and the program instance with the error label outside the scheduling range;

[0010] S5: when the program instances reserved in the scheduling range are less than the training number, supplement the pre-trained program instances meeting the training number to the scheduling range;

[0011] S6: schedule the program instances in the supplemented scheduling range to the resource devices in the scheduling range for distributed training to complete the fault-tolerant rescheduling.

[0012] According to the fault-tolerant rescheduling method for the distributed training scene provided by the application, the hardware device in step S3 includes a GPU processor and an IB card.

[0013] According to the fault-tolerant rescheduling method for the distributed training scene provided by the application, the error label in step S3 includes an error keyword and an error code value.

[0014] According to the fault-tolerant rescheduling method for the distributed training scene provided by the application, the error keyword and the error code value can be configured by a regular filtering method.

[0015] According to the fault-tolerant rescheduling method for the distributed training scene provided by the application, in step S3, when the hardware device fault error judgment and the program instance error judgment are performed, the priority of the kernel log is higher than that of the training log.

[0016] According to the fault-tolerant rescheduling method for the distributed training scene provided by the application, step S5 further includes:

[0017] S51: when the program instances reserved in the scheduling range are greater than or equal to the training number, the program instances in the scheduling range are not supplemented.

[0018] The application also provides a fault-tolerant rescheduling device for a distributed training scene, which comprises:

[0019] The pre-scheduling module is used for presetting a training number and scheduling training program instances, and scheduling the program instances to resource devices for distributed training.

[0020] The monitoring module is used for monitoring the kernel log of the resource device for distributed training and the training log of the program instance.

[0021] The error monitoring module is used for adding an error label to the hardware device when a fault error occurs in the kernel log, and is also used for checking the training log of the program instance corresponding to the hardware device with the fault error, and adding an error label to the program instance with an error in the training log.

[0022] The range supplement module is configured to set a scheduling range, and place the hardware device with the error mark and the program instance with the error mark out of the scheduling range; and is further configured to supplement the pre-trained program instance satisfying the training quantity to the scheduling range when the program instance reserved in the scheduling range is less than the training quantity.

[0023] The rescheduling module is configured to schedule the program instance in the supplemented scheduling range to the resource device in the scheduling range for distributed training to complete fault-tolerant rescheduling.

[0024] The error monitoring module further comprises:

[0025] The log screening unit is configured to screen the hardware device information with the fault error in the kernel log, and further screen whether the training log of the program instance corresponding to the hardware device with the fault error is error.

[0026] The error marking unit is configured to mark the error of the error device and the error program instance screened by the log screening unit.

[0027] The application further provides a fault-tolerant rescheduling device for a distributed training scene, comprising:

[0028] A memory and at least one processor, wherein the memory stores instructions;

[0029] The at least one processor invokes the instructions in the memory, so that the document database auditing device executes the fault-tolerant rescheduling method for a distributed training scene according to any one of the above.

[0030] The application further provides a computer readable storage medium, wherein the computer readable storage medium stores instructions, and the instructions are executed by a processor to implement the fault-tolerant rescheduling method for a distributed training scene according to any one of the above.

[0031] The application further provides a fault-tolerant rescheduling method, device, equipment and storage medium for a distributed training scene, wherein when a hardware fails in a distributed training scene, the hardware failure is identified by comparing a kernel log and a training instance log, and automatic training instance fault-tolerant rescheduling is performed, so that the uncertain factors such as fault-tolerant training program coding standardization and error identification effectiveness are greatly reduced, and at the same time, the failure device is given a prohibited scheduling label to avoid repeated scheduling to the failure device, and the success rate of fault-tolerant scheduling is effectively improved.

[0032] Additional aspects and advantages of the application will be made apparent by the following description. BRIEF DESCRIPTION OF DRAWINGS

[0033] In order to more clearly illustrate the technical solutions of the present application or prior art, the drawings required to be used in the embodiments or prior art description will be briefly introduced as follows. Obviously, the drawings in the following description are some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor on the basis of these drawings.

[0034] Fig. 1 is a flow chart of a fault-tolerant rescheduling method of a distributed training scene provided by the present application;

[0035] Fig. 2 is a structural schematic diagram of a fault-tolerant rescheduling device of a distributed training scene provided by the present application.

[0036] Reference signs:

[0037] 100, pre-scheduling module; 200, monitoring module; 300, error monitoring module; 310, log screening unit; 320, error marking unit; 400, range supplementing module; 500, rescheduling module. DETAILED DESCRIPTION

[0038] In order to make the purpose, technical solutions and advantages of the present application more clear, the technical solutions in the present application will be clearly and completely described below in combination with the drawings in the present application. Obviously, the described embodiments are some embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application. The following embodiments are used to illustrate the present application, but cannot be used to limit the scope of the present application.

[0039] In the description of the embodiments of the present application, it should be noted that the orientations or positional relationships indicated by the terms “center”, “longitudinal”, “transverse”, “upper”, “lower”, “front”, “back”, “left”, “right”, “vertical”, “horizontal”, “top”, “bottom”, “inner”, “outer” and the like are based on the orientations or positional relationships shown in the drawings, and are only for the convenience of describing the embodiments of the present application and simplifying the description, and therefore cannot be understood as indicating or implying that the devices or elements indicated thereby must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as limiting the embodiments of the present application. In addition, the terms “first”, “second”, “third” are only for the purpose of description, and cannot be understood as indicating or implying relative importance.

[0040] In the description of the embodiments of the present application, it should be noted that, unless otherwise explicitly specified and limited, the terms "connected", "connected" should be understood in a broad sense, for example, it can be fixedly connected, or it can be detachably connected, or integrally connected, it can be mechanically connected, or it can be electrically connected, it can be directly connected, or it can be indirectly connected through an intermediate medium. For those skilled in the art, the specific meaning of the above terms in the embodiments of the present application can be understood according to the specific circumstances.

[0041] In the embodiments of the present application, unless otherwise explicitly specified and limited, the first feature is "on" or "under" the second feature, which can be that the first and second features are in direct contact, or the first and second features are in indirect contact through an intermediate medium. Moreover, the first feature "above", "over" and "on" the second feature can be that the first feature is directly above or obliquely above the second feature, or only indicates that the horizontal height of the first feature is higher than that of the second feature. The first feature "below", "under" and "under" the second feature can be that the first feature is directly below or obliquely below the second feature, or only indicates that the horizontal height of the first feature is less than that of the second feature.

[0042] In the description of the present specification, the description of the terms "one embodiment", "some embodiments", "example", "specific example" or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the embodiments of the present application. In the present specification, the illustrative description of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art can combine and combine different embodiments or examples described in the present specification and the features of different embodiments or examples without contradiction.

[0043] In order to better understand the present application, the following explains the proper nouns mentioned in the embodiments of the present application.

[0044] Distributed training: the essence is distributed computing, that is, using a cluster composed of multiple machines to divide a huge and complex problem into multiple small and simple problems and solve them in parallel, and combine the results of small problems into the final result.

[0045] The embodiments of the present application are described below in conjunction with Figs. 1-2 The embodiments of the present application are described below in conjunction with

[0046] The present application provides a fault-tolerant rescheduling method for a distributed training scenario, comprising:

[0047] S1: presetting the number of training and scheduling training program instances, and scheduling the program instances to resource devices for distributed training;

[0048] S2: listen to the kernel log of the resource device performing distributed training and the training log of the program instance;

[0049] Further, first, train the scheduler, train the program instance to the device with resources according to the set number, and start the training program in the instance, listen to the kernel log of the training device and the training log of the program instance.

[0050] S3: when the hardware device information with fault error appears in the kernel log, add an error mark to the hardware device with fault error; check the training log of the program instance corresponding to the hardware device with fault error, and add an error mark to the program instance with error in the training log;

[0051] Further, when the GPU error, communication card error and other hardware fault errors in the kernel log are filtered out, the device is marked with an error, and it is checked whether the training program instance on the device also has an error, and if so, the training program instance is marked with an error.

[0052] In step S3, the hardware device includes a GPU processor and an IB card.

[0053] Further, the contents monitored by the present application include but are not limited to GPU errors, IB card communication errors and other hardware fault errors in the kernel log.

[0054] In step S3, the error mark includes an error keyword and an error code value.

[0055] In step S3, the error keyword and the error code value can be configured by a regular filtering method.

[0056] In step S3, when the hardware device fault error judgment and the program instance error judgment are performed, the priority of the kernel log is higher than that of the training log.

[0057] Further, the reason why the priority of the kernel log is higher than that of the training log is that the kernel log standard specification is beneficial for checking, so the kernel log is mainly filtered, and the training program instance log is auxiliary.

[0058] Further, the device state also needs to be listened to, after the error mark is found, the device where the instance is located is marked with a prohibited scheduling tag to exclude the node from the scheduling range, and the training instances on the error device are traversed to delete the training instances with error marks.

[0059] S4: set the scheduling range, and place the hardware device with the error mark and the program instance with the error mark outside the scheduling range;

[0060] S5: when the program instances reserved in the scheduling range are less than the training number, supplementing pre-trained program instances meeting the training number to the scheduling range;

[0061] S5 further comprises:

[0062] S51: when the program instances reserved in the scheduling range are greater than or equal to the training number, no supplement is made to the program instances in the scheduling range.

[0063] S6: scheduling the supplemented program instances in the scheduling range to resource devices in the scheduling range for distributed training to complete fault-tolerant rescheduling.

[0064] Further, when the problem training program instances are deleted, the training scheduler checks that the number of training program instances is less than a set value, schedules new training program instances to resource devices, excludes devices prohibited from being scheduled, starts the training program of the instance after successful scheduling, and until the number of instances reaches the expected value, the fault-tolerant rescheduling is completed.

[0065] The application also provides a fault-tolerant rescheduling device for a distributed training scene, comprising:

[0066] A pre-scheduling module 100 is configured to preset a training number and schedule training program instances, and schedule the program instances to resource devices for distributed training.

[0067] A listening module 200 is configured to listen to kernel logs of the resource devices for distributed training and training logs of the program instances.

[0068] An error monitoring module 300 is configured to add an error label to a hardware device with a fault error in the kernel logs, and check training logs of program instances corresponding to the hardware device with the fault error, and add an error label to program instances with errors in the training logs.

[0069] A range supplement module 400 is configured to set a scheduling range, place hardware devices with error labels and program instances with error labels outside the scheduling range, and when the program instances reserved in the scheduling range are less than the training number, supplement pre-trained program instances meeting the training number to the scheduling range.

[0070] A rescheduling module 500 is configured to schedule the supplemented program instances in the scheduling range to resource devices in the scheduling range for distributed training to complete fault-tolerant rescheduling.

[0071] The error monitoring module further comprises:

[0072] The log screening unit 310 is configured to screen the hardware device information of the fault error in the kernel log, and screen whether the training log of the program instance corresponding to the hardware device with the fault error is error.

[0073] The error marking unit 320 is configured to mark the error of the error device and the error program instance screened by the log screening unit 310.

[0074] In the application of the fault-tolerant rescheduling device for a distributed training scenario provided by the application, first, the preset training quantity is scheduled by the pre-scheduling module, and the program instance is scheduled to the resource device for distributed training. When the distributed training is performed, the kernel log and the training log of the program instance of the resource device are listened to by the listening module. Then, the log screening unit in the error monitoring module screens the hardware device information of the fault error in the kernel log, and screens whether the training log of the program instance corresponding to the hardware device with the fault error is error. The error marking unit in the error monitoring module marks the error of the error device and the error program instance screened by the log screening unit. The scheduling range is set, and the hardware device with the error mark and the program instance with the error mark are placed outside the scheduling range by the range supplement module. When the program instance remaining in the scheduling range is less than the training quantity, the pre-trained program instance meeting the training quantity is supplemented to the scheduling range. Finally, the program instance in the scheduling range after the scheduling supplement is rescheduled to the resource device in the scheduling range by the rescheduling module for distributed training, so as to complete the fault-tolerant rescheduling.

[0075] The application further provides a fault-tolerant rescheduling device for a distributed training scenario, comprising:

[0076] a memory and at least one processor, wherein the memory stores instructions;

[0077] The at least one processor calls the instructions in the memory, so that the document database audit device performs the fault-tolerant rescheduling method for a distributed training scenario as described in any one of the above.

[0078] In some embodiments, the device can have a large difference in configuration or performance, and can include one or more processors, such as one or more processors and memories, one or more storage media, such as one or more mass storage devices, to store applications or data. Among them, the memory and the storage medium can be temporary storage or persistent storage. The program stored in the storage medium can include one or more modules, each of which can include a series of instruction operations in the device. Further, the processor can be configured to communicate with the storage medium to execute a series of instruction operations in the storage medium on the device.

[0079] The device can also include one or more power supplies, one or more wired or wireless network interfaces, one or more input / output interfaces, and / or one or more operating systems, such as Windows Server, Mac OS X, Unix, Linux, FreeBSD, etc. Those skilled in the art can understand that the device structure shown above does not constitute a limitation on the device, and can include more or fewer components than shown, or combine certain components, or different component arrangements.

[0080] The present application also provides a computer-readable storage medium having instructions stored thereon, which, when executed by a processor, implement a fault-tolerant rescheduling method for a distributed training scenario as described in any one of the above.

[0081] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the above-described system, device and unit can refer to the corresponding process in the foregoing method embodiments, which will not be described here.

[0082] The integrated unit, if implemented in the form of a software function unit and sold as an independent product or through time, can be stored in a computer-readable storage medium. Based on the understanding, the technical solutions of the present application essentially or say the part that contributes to the prior art or the whole or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, including a number of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the method described in each embodiment of the present application. The foregoing storage medium includes: a U disk, a mobile hard disk, a read-only memory (read-only memory, ROM), a random access memory (random access memory, RAM), a magnetic disk or an optical disk, and various media that can store program codes.

[0083] The application provides a fault-tolerant rescheduling method and device for a distributed training scene, equipment and a storage medium. Hardware faults are identified by comparing kernel logs and training instance logs, and automatic fault-tolerant rescheduling of training instances is performed, so that the uncertainty factors such as fault-tolerant training program coding standardization and error identification effectiveness are greatly reduced. In addition, the fault equipment is labeled with a prohibited scheduling tag to avoid repeated scheduling to the fault equipment, and the efficiency and success rate of rescheduling can be improved.

[0084] Finally, it should be noted that: the above examples are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing examples, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing examples, or make equivalent replacement for part of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A fault-tolerant rescheduling method for distributed training scenarios, characterized in that: include: S1: Preset the number of training programs and schedule training program instances, and schedule the program instances to resource devices for distributed training; S2: monitoring the kernel log of the resource device performing distributed training and the training log of the program instance; S3: When information about a hardware device with a fault error appears in the kernel log, an error mark is added to the hardware device with the fault error; Checking the training log of the program instance corresponding to the hardware device that has a fault error, and adding an error mark to the program instance that reports an error in the training log; S4: Set the scheduling range and place the hardware devices with error marks and the program instances with error marks outside the scheduling range; S5: When the number of program instances remaining in the scheduling range is less than the training quantity, add pre-trained program instances that meet the training quantity to the scheduling range; S6: Dispatching the supplemented program instances within the scheduling range to the resource devices within the scheduling range for distributed training to complete fault-tolerant rescheduling.

2. The fault-tolerant rescheduling method for a distributed training scenario according to claim 1, characterized in that: The hardware devices in step S3 include a GPU processor and an IB card.

3. The fault-tolerant rescheduling method for a distributed training scenario according to claim 1, characterized in that: The error flag in step S3 includes an error keyword and an error code value.

4. The fault-tolerant rescheduling method for a distributed training scenario according to claim 3, characterized in that: Both the error keyword and the error code value can be configured through regular filtering.

5. The fault-tolerant rescheduling method for a distributed training scenario according to claim 1, characterized in that: In step S3, when performing hardware device failure error determination and program instance error determination, the kernel log has a higher priority than the training log.

6. The fault-tolerant rescheduling method for a distributed training scenario according to claim 1, characterized in that: Step S5 further includes: S51: When the number of program instances remaining in the scheduling range is greater than or equal to the training quantity, the program instances in the scheduling range are not supplemented.

7. A fault-tolerant rescheduling device for distributed training scenarios, characterized in that: include: Pre-scheduling module: used to preset the number of training sessions and schedule training program instances, and dispatch the program instances to resource devices for distributed training; Monitoring module: used for monitoring and obtaining the kernel log of the resource device for distributed training and the training log of the program instance; An error monitoring module is configured to add an error mark to a hardware device when a hardware device with a fault error appears in the kernel log; Also used to check the training log of the program instance corresponding to the hardware device that has a fault error, and add an error mark to the program instance that reports an error in the training log; Range supplement module: used to set the scheduling range and place hardware devices with error marks and program instances with error marks outside the scheduling range; further configured to, when the number of program instances remaining in the scheduling range is less than the number of training instances, add pre-trained program instances that meet the number of training instances to the scheduling range; Rescheduling module: used to schedule the supplemented program instances within the scheduling range to the resource devices within the scheduling range for distributed training to complete fault-tolerant rescheduling.

8. The fault-tolerant rescheduling device for a distributed training scenario according to claim 7, characterized in that: The error monitoring module further includes: Log screening unit: used to screen the kernel log for hardware device information with fault errors, and also used to screen the training log of the program instance corresponding to the hardware device with fault errors for errors; Error marking unit: used to mark the error devices and error-reporting program instances obtained by the log screening unit.

9. A fault-tolerant rescheduling device for distributed training scenarios, characterized in that: include: a memory and at least one processor, wherein instructions are stored in the memory; At least one of the processors calls the instructions in the memory so that the document database audit device executes the fault-tolerant rescheduling method for a distributed training scenario as described in any one of claims 1 to 6.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores instructions, which, when executed by a processor, implement a fault-tolerant rescheduling method for a distributed training scenario as described in any one of claims 1 to 6.

Images