A model training duration determination method and apparatus

Abstract

Embodiments of the present application provide a model training duration determination method and device, relating to the technical field of training platforms, comprising: a training server obtaining a target training task for training a target image processing model based on a target sample image; determining, based on a data amount of the target sample image, a first correspondence relationship between a preset data amount and an acquisition duration, an acquisition duration required for acquiring the target sample image, obtaining a data preparation duration of the target training task; determining, based on a target training parameter of the target training task, a second correspondence relationship between a preset training parameter group and an execution duration, a task execution duration of the target training task; calculating, based on the data preparation duration and the task execution duration of the target training task, a total training duration of the target training task, and sending the total training duration of the target training task to a terminal. The terminal displays the total training duration of the target training task, so that the user knows the time when the target image processing model is trained, and the user experience is improved.

Description

Technical Field

[0001] This application relates to the field of model training platform technology, and in particular to a method and apparatus for determining model training duration. Background Technology

[0002] Before performing image recognition, object detection, and other processing based on an image processing model, the model can be trained. Users can submit training tasks for the image processing model to an AI (Artificial Intelligence) training platform via their terminals. The AI ​​training platform can execute the training tasks and complete the training of the image processing model. Due to the limited number of graphics cards on the AI ​​training platform, each graphics card can only execute one training task at a time. When the number of training tasks exceeds the number of graphics cards on the AI ​​training platform, the training tasks need to be queued.

[0003] In related technologies, after a training task is submitted to an AI training platform, the terminal can display a list page of training tasks, which shows the current status of the training tasks. The status of the training tasks includes four states: waiting in the queue, preparing data, training, and transforming.

[0004] However, the terminal can only display the status of the training task. The relevant technologies do not provide a corresponding method to determine the training duration of the training task, which makes it impossible for users to know when the image processing model will be trained, thus reducing the user experience. Summary of the Invention

[0005] The purpose of this application is to provide a method and apparatus for determining model training duration, so as to determine the training duration of a training task and display the training duration of the training task to the user, allowing the user to know exactly when the image processing model will complete training, thereby improving the user experience. The specific technical solution is as follows:

[0006] Firstly, to achieve the above objectives, embodiments of this application provide a method for determining model training duration. The method is applied to a training server in a model training platform, which further includes a terminal. The method includes:

[0007] Obtain the target training task; wherein, the target training task is used to train the target image processing model based on the target sample image;

[0008] Based on the data volume of the target sample image and a preset first correspondence between the data volume and the acquisition time, the acquisition time required to acquire the target sample image is determined, and the data preparation time for the target training task is obtained; wherein, the first correspondence is determined based on the data volume of historical sample images corresponding to historical training tasks executed by the training server and the acquisition time for acquiring the historical sample images;

[0009] Based on the target training parameters of the target training task and the second correspondence between the preset training parameter set and the execution time, the task execution time of the target training task is determined; wherein, the target training parameters include: the algorithm parameters of the target image processing model, the image parameters of the target sample image, and the execution parameters of the target training task; the second correspondence is determined based on the training parameters and task execution time of the historical training tasks;

[0010] Based on the data preparation time and task execution time of the target training task, calculate the total training time of the target training task;

[0011] The total training time of the target training task is sent to the terminal so that the terminal displays the total training time of the target training task.

[0012] Optionally, before calculating the total training time of the target training task based on the data preparation time and task execution time of the target training task, the method further includes:

[0013] Based on a preset task scheduling strategy, the graphics card in the training server used to execute the target training task is determined as the target graphics card;

[0014] The training task that the target graphics card needs to perform before executing the target training task is determined and designated as the first training task;

[0015] From the first training task, determine the first training task currently being executed by the target graphics card as the second training task, and determine the first training task to be executed by the target graphics card as the third training task;

[0016] The remaining execution time of the second training task, the data preparation time of the third training task, and the sum of the task execution time are calculated to obtain the waiting time of the target training task.

[0017] The calculation of the total training time for the target training task based on the data preparation time and task execution time includes:

[0018] The total training time of the target training task is obtained by summing the waiting time, data preparation time, and task execution time of the target training task.

[0019] Optionally, before calculating the sum of the remaining execution time of the second training task, the data preparation time of the third training task, and the task execution time to obtain the waiting time of the target training task, the method further includes:

[0020] The total number of iterations for the second training task is calculated based on the number of first sample images corresponding to the second training task, the batch size of the second training task, and the batch iteration number of the second training task.

[0021] The difference between the total number of iterations for the second training task and the number of iterations already performed in the second training task is calculated to obtain the remaining number of iterations for the second training task.

[0022] The remaining execution time of the second training task is calculated based on the single execution time of the iterations already performed in the second training task and the remaining number of iterations of the second training task.

[0023] Optionally, determining the task execution time of the target training task based on the target training parameters of the target training task and the second correspondence between the preset training parameter set and the execution time includes:

[0024] In the second correspondence between the preset training parameter group and the execution time, the execution time corresponding to the training parameter group to which the target training parameter belongs is determined as the estimated single execution time of the target training task for one iteration calculation;

[0025] The total number of iterations for the target training task is calculated based on the number of target sample images, the batch size of the target training task, and the batch iteration number of the target training task.

[0026] The execution time of the target training task is obtained by multiplying the total number of iterations of the target training task by the estimated single execution time of the target training task.

[0027] Optionally, the algorithm parameters of the target image processing model include: the algorithm type of the target image processing model; the image parameters of the target sample image include: the image type, width, and height of the target sample image; and the execution parameters of the target training task include: the batch size of the target training task.

[0028] One of the training parameter groups in the second correspondence includes: algorithm type, image type, preset image width range, preset image height range, and batch size;

[0029] The target training parameters belong to a training parameter group, which means that the algorithm type in the training parameter group is the same as the algorithm type of the target image processing model, the image type is the same as the image type of the target sample image, the batch size is the same as the batch size of the target training task, the preset image width range includes the image width of the target sample image, and the preset image height range includes the image height of the target sample image.

[0030] Optionally, before determining the execution time of the target training task based on the target training parameters of the target training task and the second correspondence between the preset training parameter set and the execution time, the method further includes:

[0031] For each historical training task, the ratio of the task execution time to the total number of iterations of the historical training task is calculated to obtain the single execution time of one iteration of the historical training task.

[0032] Historical training tasks whose training parameters belong to the same training parameter group are grouped together to obtain multiple first training task groups; each training parameter group includes: algorithm type, image type, preset image width range, preset image height range, and batch size;

[0033] For each first training task group, calculate the average execution time of each historical training task in the first training task group to obtain the execution time of the training parameter group corresponding to the first training task group.

[0034] Record the correspondence between each training parameter set and the execution duration to obtain the second correspondence between the training parameter set and the execution duration.

[0035] Optionally, after calculating the total training time of the target training task based on the data preparation time and task execution time of the target training task, the method further includes:

[0036] After the target training task is completed, the ratio of the task execution time to the total number of iterations of the target training task is calculated to obtain the actual single execution time of the target training task for one iteration.

[0037] The second correspondence is updated based on the target training parameters and the actual execution time of one iteration of the target training task.

[0038] Optionally, determining the acquisition time required to acquire the target sample image based on the data volume of the target sample image and a preset first correspondence between the data volume and the acquisition time, thereby obtaining the data preparation time for the target training task, includes:

[0039] Calculate the ratio of the total data volume to the number of target sample images to obtain the average data volume of each target sample image;

[0040] In the first correspondence between the preset data volume and the acquisition time, the acquisition time corresponding to the data volume with the smallest difference from the calculated average data volume is determined, and the estimated acquisition time for acquiring a target sample image is obtained.

[0041] Calculate the product of the estimated acquisition time of a target sample image and the number of target sample images to obtain the total acquisition time required to acquire each target sample image, which is used as the data preparation time for the target training task.

[0042] Optionally, before determining the acquisition time required to acquire the target sample image based on the data volume of the target sample image and a preset first correspondence between the data volume and the acquisition time, and obtaining the data preparation time for the target training task, the method further includes:

[0043] For each historical training task, the ratio of the total data volume to the number of historical sample images corresponding to that historical training task is calculated to obtain the average data volume of each historical sample image.

[0044] The average acquisition time of each historical sample image is obtained by calculating the ratio of the total acquisition time to the number of historical sample images when performing the historical training task.

[0045] Historical training tasks with the same average data volume for each historical sample image are grouped together to obtain multiple second training task groups.

[0046] For each second training task group, calculate the mean of the average acquisition time corresponding to each historical training task in the second training task group, and obtain the average acquisition time corresponding to the average data volume of the second training task group.

[0047] Record the correspondence between the average data volume and the average acquisition time to obtain the first correspondence between data volume and acquisition time.

[0048] Optionally, after calculating the total training time of the target training task based on the data preparation time and task execution time of the target training task, the method further includes:

[0049] After the target training task is completed, the ratio of the total acquisition time of each target sample image to the number of each target sample image is calculated to obtain the average acquisition time of each target sample image.

[0050] The first correspondence is updated based on the average data volume and average acquisition time of each target sample image.

[0051] Secondly, to achieve the above objectives, embodiments of this application provide a model training platform, which includes: a training server and a terminal; wherein,

[0052] The training server is used to acquire a target training task; wherein the target training task is used to train a target image processing model based on target sample images; based on the data volume of the target sample images and a preset first correspondence between data volume and acquisition time, the acquisition time required to acquire the target sample images is determined, thus obtaining the data preparation time of the target training task; wherein the first correspondence is determined based on the data volume of historical sample images corresponding to historical training tasks executed by the training server and the acquisition time of acquiring the historical sample images; based on the target training parameters of the target training task and a preset second correspondence between training parameter groups and execution time, the task execution time of the target training task is determined; wherein the target training parameters include: algorithm parameters of the target image processing model, image parameters of the target sample images, and execution parameters of the target training task; the second correspondence is determined based on the training parameters and task execution time of the historical training tasks; based on the data preparation time and task execution time of the target training task, the total training time of the target training task is calculated; and the total training time of the target training task is sent to the terminal.

[0053] The terminal is used to display the total training time of the received target training task.

[0054] Thirdly, to achieve the above objectives, embodiments of this application provide a model training duration determination device. The device is applied to a training server in a model training platform, which further includes a terminal. The device comprises:

[0055] The target training task acquisition module is used to acquire the target training task; wherein, the target training task is used to train the target image processing model based on the target sample image;

[0056] The data preparation time determination module is used to determine the acquisition time required to acquire the target sample image based on the data volume of the target sample image and a preset first correspondence between the data volume and the acquisition time, thereby obtaining the data preparation time of the target training task; wherein, the first correspondence is determined based on the data volume of historical sample images corresponding to historical training tasks already executed by the training server and the acquisition time of acquiring the historical sample images;

[0057] The task execution duration determination module is used to determine the task execution duration of the target training task based on the target training parameters of the target training task and a second correspondence between a preset training parameter set and the execution duration; wherein, the target training parameters include: the algorithm parameters of the target image processing model, the image parameters of the target sample image, and the execution parameters of the target training task; the second correspondence is determined based on the training parameters and task execution duration of the historical training tasks;

[0058] The total training time determination module is used to calculate the total training time of the target training task based on the data preparation time and task execution time of the target training task;

[0059] The total training time display module is used to send the total training time of the target training task to the terminal so that the terminal can display the total training time of the target training task.

[0060] Optionally, the device further includes:

[0061] The target graphics card determination module is used to determine the graphics card in the training server used to execute the target training task as the target graphics card before the total training time determination module executes the data preparation time and task execution time based on the target training task and calculates the total training time of the target training task.

[0062] The first training task determination module is used to determine the training task that the target graphics card needs to perform before executing the target training task, as the first training task;

[0063] The second training task determination module is used to determine, from the first training task, the first training task currently being executed by the target graphics card as the second training task, and to determine the first training task to be executed by the target graphics card as the third training task.

[0064] The waiting time determination module is used to calculate the sum of the remaining execution time of the second training task, the data preparation time of the third training task, and the task execution time to obtain the waiting time of the target training task;

[0065] The total training time determination module is specifically used to calculate the sum of the waiting time, data preparation time and task execution time of the target training task to obtain the total training time of the target training task.

[0066] Optionally, the device further includes:

[0067] The total iteration count determination module is used to calculate the total number of iterations of the second training task before the waiting time determination module calculates the remaining execution time of the second training task, the data preparation time of the third training task, and the sum of the task execution time to obtain the waiting time of the target training task. This is done based on the number of first sample images corresponding to the second training task, the batch size of the second training task, and the batch iteration count of the second training task.

[0068] The remaining iteration count determination module is used to calculate the difference between the total number of iterations of the second training task and the number of iterations calculated by the iterations already performed in the second training task, so as to obtain the remaining number of iterations of the second training task;

[0069] The remaining execution time determination module is used to calculate the remaining execution time of the second training task based on the single execution time calculated from the iterations already executed by the second training task and the remaining number of iterations of the second training task.

[0070] Optionally, the task execution time determination module is specifically used to determine the execution time corresponding to the training parameter group to which the target training parameter belongs in the second correspondence between the preset training parameter group and the execution time, as the estimated single execution time of the target training task for one iteration calculation;

[0071] The total number of iterations for the target training task is calculated based on the number of target sample images, the batch size of the target training task, and the batch iteration number of the target training task.

[0072] The execution time of the target training task is obtained by multiplying the total number of iterations of the target training task by the estimated single execution time of the target training task.

[0073] Optionally, the algorithm parameters of the target image processing model include: the algorithm type of the target image processing model; the image parameters of the target sample image include: the image type, width, and height of the target sample image; and the execution parameters of the target training task include: the batch size of the target training task.

[0074] One of the training parameter groups in the second correspondence includes: algorithm type, image type, preset image width range, preset image height range, and batch size;

[0075] The target training parameters belong to a training parameter group, which means that the algorithm type in the training parameter group is the same as the algorithm type of the target image processing model, the image type is the same as the image type of the target sample image, the batch size is the same as the batch size of the target training task, the preset image width range includes the image width of the target sample image, and the preset image height range includes the image height of the target sample image.

[0076] Optionally, the device further includes:

[0077] The first single execution duration determination module is used to determine the task execution duration of the target training task based on the target training parameters of the target training task and the second correspondence between the preset training parameter group and the execution duration before the task execution duration determination module executes the task execution duration of the target training task. For each historical training task, the module calculates the ratio of the task execution duration of the historical training task to the total number of iterations of the historical training task to obtain the single execution duration of the historical training task for one iteration calculation.

[0078] The first training task grouping module is used to group historical training tasks whose training parameters belong to the same training parameter group into a group, resulting in multiple first training task groups; wherein, a training parameter group includes: algorithm type, image type, preset image width range, preset image height range, and batch size;

[0079] The second single execution duration determination module is used to calculate the average single execution duration of each historical training task in each first training task group, so as to obtain the single execution duration corresponding to the training parameter group corresponding to the first training task group.

[0080] The second correspondence determination module is used to record the correspondence between each training parameter group and the single execution duration, and obtain the second correspondence between the training parameter group and the execution duration.

[0081] Optionally, the device further includes:

[0082] The actual single execution time determination module is used to calculate the total training time of the target training task based on the data preparation time and task execution time of the target training task after the total training time determination module has executed the target training task. After the target training task has been executed, the module calculates the ratio of the task execution time of the target training task to the total number of iterations of the target training task to obtain the actual single execution time of the target training task for one iteration.

[0083] The second correspondence update module is used to update the second correspondence based on the target training parameters and the actual execution time of one iteration of the target training task.

[0084] Optionally, the data preparation time determination module is specifically used to calculate the ratio of the total data volume to the number of target sample images to obtain the average data volume of each target sample image.

[0085] In the first correspondence between the preset data volume and the acquisition time, the acquisition time corresponding to the data volume with the smallest difference from the calculated average data volume is determined, and the estimated acquisition time for acquiring a target sample image is obtained.

[0086] Calculate the product of the estimated acquisition time of a target sample image and the number of target sample images to obtain the total acquisition time required to acquire each target sample image, which is used as the data preparation time for the target training task.

[0087] Optionally, the device further includes:

[0088] The average data volume determination module is used to determine the acquisition time required to acquire the target sample image based on the data volume of the target sample image and the first correspondence between the preset data volume and the acquisition time in the data preparation time determination module, and to obtain the data preparation time of the target training task before the data preparation time determination module executes the following: for each historical training task, the module calculates the ratio of the total data volume to the number of historical sample images corresponding to the historical training task, and obtains the average data volume of each historical sample image.

[0089] The first average acquisition time determination module is used to calculate the ratio of the total acquisition time of each historical sample image to the number of each historical sample image when performing the historical training task, so as to obtain the average acquisition time of each historical sample image.

[0090] The second training task grouping module is used to divide historical training tasks with the same average data volume of each historical sample image into a group, resulting in multiple second training task groups.

[0091] The second average acquisition time determination module is used to calculate the mean of the average acquisition time of each historical training task in the second training task group for each second training task group, so as to obtain the average acquisition time corresponding to the average data volume of the second training task group.

[0092] The first correspondence determination module is used to record the correspondence between each average data volume and the average acquisition time, and to obtain the first correspondence between data volume and acquisition time.

[0093] Optionally, the device includes:

[0094] The third average acquisition time determination module is used to calculate the average acquisition time of each target sample image after the total training time determination module performs the data preparation time and task execution time based on the target training task and calculates the total training time of the target training task. After the target training task is completed, the module calculates the ratio of the total acquisition time of each target sample image to the number of each target sample image during the execution of the target training task, and obtains the average acquisition time of each target sample image.

[0095] The first correspondence update module is used to update the first correspondence based on the average data volume and average acquisition time of each target sample image.

[0096] This application also provides a training server, including:

[0097] Memory, used to store computer programs;

[0098] When a processor executes a program stored in memory, it implements any of the above-described methods for determining the model training duration.

[0099] This application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements any of the above-described methods for determining model training duration.

[0100] This application also provides a computer program product containing instructions that, when run on a computer, cause the computer to execute any of the above-described methods for determining model training duration.

[0101] Beneficial effects of the embodiments in this application:

[0102] This application provides a method for determining model training duration. A training server acquires a target training task. The target training task is used to train a target image processing model based on target sample images. Based on the data volume of the target sample images and a preset first correspondence between data volume and acquisition duration, the acquisition duration required to acquire the target sample images is determined, resulting in the data preparation duration for the target training task. The first correspondence is determined based on the data volume of historical sample images corresponding to historical training tasks already executed by the training server and the acquisition duration of those historical sample images. Based on the target training parameters of the target training task and a preset second correspondence between the training parameter set and execution duration, the task execution duration of the target training task is determined. The target training parameters include: algorithm parameters of the target image processing model, image parameters of the target sample images, and execution parameters of the target training task. The second correspondence is determined based on the training parameters and task execution duration of historical training tasks. Based on the data preparation duration and task execution duration of the target training task, the total training duration of the target training task is calculated. The total training duration of the target training task is sent to the terminal. The terminal displays the total training duration of the target training task.

[0103] Based on the above processing, the data preparation time for the target training task can be determined based on the data volume of the target sample image and a preset first correspondence between data volume and acquisition time. Furthermore, the task execution time for the target training task can be determined based on the target training parameters and a preset second correspondence between training parameter sets and execution time. Then, based on the data preparation time and task execution time of the target training task, the total training time of the target training task is calculated and displayed to the user. This allows the user to know exactly when the target image processing model will complete training, improving the user experience.

[0104] Of course, implementing any product or method of this application does not necessarily require achieving all of the advantages described above at the same time. Attached Figure Description

[0105] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other embodiments can be obtained based on these drawings.

[0106] Figure 1 A flowchart illustrating the first method for determining model training duration provided in this application embodiment;

[0107] Figure 2 A structural diagram of a model training platform provided in an embodiment of this application;

[0108] Figure 3 A flowchart illustrating the second method for determining model training duration provided in this application embodiment;

[0109] Figure 4 A flowchart illustrating the third method for determining model training duration provided in this application embodiment;

[0110] Figure 5 A flowchart illustrating the fourth method for determining model training duration provided in this application embodiment;

[0111] Figure 6 A flowchart illustrating the fifth method for determining model training duration provided in this application embodiment;

[0112] Figure 7 A flowchart illustrating the sixth method for determining model training duration provided in this application embodiment;

[0113] Figure 8 A flowchart illustrating the seventh method for determining model training duration provided in this application embodiment;

[0114] Figure 9 A schematic diagram of a terminal display interface provided in an embodiment of this application;

[0115] Figure 10 A structural diagram of a model training duration determination device provided in an embodiment of this application;

[0116] Figure 11 This is a structural diagram of a first type of training server provided in an embodiment of this application. Detailed Implementation

[0117] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art based on this application are within the scope of protection of this application.

[0118] In related technologies, no corresponding method is provided to determine the training duration of a training task. The terminal can only display the status of the training task, which makes it impossible for users to know when the image processing model will be trained, thus reducing the user experience.

[0119] To address the aforementioned issues, this application provides a method for determining model training duration. This method is applied to a training server in a model training platform. The training server acquires a target training task for training a target image processing model based on target sample images. Based on the data volume of the target sample images and a preset first correspondence between data volume and acquisition duration, the acquisition duration required to acquire the target sample images is determined, resulting in the data preparation duration for the target training task. Based on the target training parameters of the target training task and a preset second correspondence between training parameter sets and execution duration, the task execution duration of the target training task is determined. Based on the data preparation duration and task execution duration of the target training task, the total training duration of the target training task is calculated. The total training duration of the target training task is then sent to the terminal. The terminal displays the total training duration of the target training task, allowing users to know exactly when the target image processing model will complete training, thus improving the user experience.

[0120] See Figure 1 , Figure 1 A flowchart of a method for determining model training duration provided in this application embodiment. The method is applied to a training server in a model training platform, which also includes a terminal. The method may include the following steps:

[0121] S101: Obtain the target training task.

[0122] The target training task is used to train the target image processing model based on the target sample image.

[0123] S102: Based on the amount of data in the target sample image and the first correspondence between the preset amount of data and the acquisition time, determine the acquisition time required to acquire the target sample image, and obtain the data preparation time for the target training task.

[0124] The first correspondence is determined based on the amount of historical sample images corresponding to the historical training tasks executed by the training server and the acquisition time of the historical sample images.

[0125] S103: Based on the target training parameters of the target training task and the second correspondence between the preset training parameter set and the execution time, determine the task execution time of the target training task.

[0126] The target training parameters include: algorithm parameters of the target image processing model, image parameters of the target sample image, and execution parameters of the target training task. The second correspondence is determined based on the training parameters and task execution time of historical training tasks.

[0127] S104: Calculate the total training time of the target training task based on the data preparation time and task execution time of the target training task.

[0128] S105: Send the total training time of the target training task to the terminal so that the terminal can display the total training time of the target training task.

[0129] Based on the model training duration determination method provided in this application, the data preparation duration of the target training task can be determined based on the data volume of the target sample image and a preset first correspondence between the data volume and the acquisition duration. Furthermore, the task execution duration of the target training task can be determined based on the target training parameters of the target training task and a preset second correspondence between the training parameters and the execution duration. Then, based on the data preparation duration and task execution duration of the target training task, the total training duration of the target training task is calculated and displayed to the user, allowing the user to know exactly when the target image processing model will complete training, thus improving the user experience.

[0130] Regarding step S101, the model training platform is a one-stop AI open platform providing functions such as data management, data annotation, model training, model validation, and model conversion. See also... Figure 2 , Figure 2 This is a structural diagram of a model training platform provided in an embodiment of this application. The model training platform may include: a storage server 201, a training server 202, and a terminal 203. The training server 202 communicates with the storage server 201 and the terminal 203 respectively. The storage server 201 is used to store sample images for training an image processing model. The training server 202 is used to execute training tasks for training the image processing model based on the sample images. The terminal 203 is used to submit training tasks to the training server and display relevant information about the training tasks.

[0131] The target training task can be any training task submitted by the terminal to the training server. The target training task is used to train the target image processing model based on the target sample image. The target image processing model can be any one of the following: target detection model, image segmentation model, image classification model, and image recognition model.

[0132] The target sample images are stored on a storage server. Before executing the target training task, the training server needs to retrieve the target sample images from the storage server. Accordingly, the total training time of the target training task includes: the data preparation time for retrieving the target sample images from the storage server when the training server starts executing the target training task, and the task execution time from the start of executing the target training task after the training server retrieves the target sample images until the completion of the target training task. The data preparation time is the time for the training server to download the target sample images from the database and storage server. This acquisition time depends on the database query speed, network speed, and the size of the target sample images.

[0133] Regarding step S102, in one implementation, each data quantity in the first correspondence is: the total data quantity of each historical sample image corresponding to a historical training task, and the acquisition time corresponding to this data quantity is: the total acquisition time of acquiring each historical sample image when executing the historical training task.

[0134] For each historical training task, the training server obtains the total amount of historical sample images corresponding to that historical training task, as well as the total acquisition time for each historical sample image during the execution of that historical training task. Then, the average of the total acquisition times for all historical training tasks with the same total amount of historical sample images is calculated as the total acquisition time corresponding to that total amount of data, and the correspondence between each total amount of data and the total acquisition time is recorded to obtain the first correspondence between data amount and acquisition time.

[0135] Accordingly, the training server can determine the total acquisition time corresponding to the data volume with the smallest difference from the total data volume of each target sample image in the first correspondence between the preset data volume and acquisition time, and use it as the total acquisition time required to acquire each target sample image, thus obtaining the data preparation time for the target training task.

[0136] In another implementation, the data volume of each data point in the first correspondence is: the average data volume of each historical sample image corresponding to a historical training task, and the acquisition time corresponding to this data volume is: the average acquisition time of acquiring each historical sample image when executing the historical training task.

[0137] Correspondingly, in Figure 1 Based on this, see Figure 3 Step S102 may include the following steps:

[0138] S1021: Calculate the ratio of the total data volume to the number of target sample images to obtain the average data volume of each target sample image.

[0139] S1022: In the first correspondence between the preset data volume and the acquisition time, determine the acquisition time corresponding to the data volume with the smallest difference from the calculated average data volume, and obtain the estimated acquisition time for acquiring a target sample image.

[0140] S1023: Calculate the product of the estimated acquisition time of a target sample image and the number of target sample images to obtain the total acquisition time required to acquire each target sample image, which is used as the data preparation time for the target training task.

[0141] The training server can calculate the ratio of the total data volume to the number of target sample images to obtain the average data volume of each target sample image, which is the average data volume of a single target sample image. Then, the training server can determine the data volume with the smallest difference from the average data volume of each target sample image from a preset first correspondence between data volume and acquisition time. Since this data volume has the smallest difference from the average data volume of each target sample image, meaning it is closest to the data volume of a single target sample image, the acquisition time for acquiring a single target sample image is also relatively close to the acquisition time for acquiring historical sample images with that data volume. Therefore, the training server can use the acquisition time corresponding to the data volume with the smallest difference from the average data volume of each target sample image as the estimated acquisition time for acquiring a single target sample image. The training server calculates the product of the estimated acquisition time for acquiring a single target sample image and the number of target sample images to obtain the total acquisition time required for the training server to acquire each target sample image, which is used as the data preparation time for the target training task.

[0142] Based on the above processing, the estimated acquisition time for a target sample image can be determined. Based on the estimated acquisition time for a target sample image and the number of each sample image, the total acquisition time for each target sample image can be determined. The acquisition time for each sample image will also be different when the number of sample images is different. Compared with directly determining the total acquisition time for each target sample image, the accuracy of the determined total acquisition time for each target sample image can be improved.

[0143] In some embodiments, prior to step S102, the method may further include the following steps:

[0144] Step 1: For each historical training task, calculate the ratio of the total data volume to the number of historical sample images corresponding to that historical training task, and obtain the average data volume of each historical sample image.

[0145] Step 2: Calculate the ratio of the total acquisition time of each historical sample image to the number of historical sample images when performing the historical training task, and obtain the average acquisition time of each historical sample image.

[0146] Step 3: Divide historical training tasks with the same average data volume of historical sample images into a group to obtain multiple second training task groups.

[0147] Step 4: For each second training task group, calculate the average acquisition time of each historical training task in the second training task group, and obtain the average acquisition time corresponding to the average data volume of the second training task group.

[0148] Step 5: Record the correspondence between the average data volume and the average acquisition time to obtain the first correspondence between data volume and acquisition time.

[0149] For example, for each historical training task, the training server calculates the average data size of each historical sample image corresponding to that historical training task based on the following formula: Average size of a single image = Total size of dataset / Number of images in dataset. Wherein, the average size of a single image is the data size of one historical sample image, i.e., the average data size of all historical sample images; the total size of the dataset is the total data size of all historical sample images; and the number of images in the dataset is the number of historical sample images.

[0150] Then, the training server calculates the average acquisition time for each historical sample image when executing the historical training task based on the following formula: Single image preparation time = Training task data preparation time / Number of dataset images. Where, single image preparation time is the acquisition time for acquiring one historical sample image, i.e., the average acquisition time for all historical sample images; training task data preparation time is the total acquisition time for acquiring all historical sample images when executing the historical training task; and the number of dataset images is the number of historical sample images.

[0151] The training server can group historical training tasks with the same average data volume for each historical sample image into multiple second training task groups. Then, it calculates the mean of the average acquisition time for each historical training task within each second training task group, obtaining the average acquisition time corresponding to the average data volume of that second training task group. Furthermore, by recording the correspondence between each average data volume and average acquisition time, a first correspondence between data volume and acquisition time can be obtained.

[0152] For example, see Table 1, which is a table showing the correspondence between data volume and acquisition time provided in an embodiment of this application. In Table 1, image size is the average data volume of each historical sample image corresponding to a historical training task; preparation time is the average acquisition time for acquiring one historical sample image.

[0153] Table 1

[0154] Image size Preparation time 200K t1 500K t2 1M t3 2M t4 3M t5 5M t6 8M t7 10M t8 15M t9

[0155] For example, if the average data size of each target sample image is 2M, the estimated acquisition time for acquiring one target sample image can be obtained from Table 1 as t4; if the average data size of each target sample image is 6M, the estimated acquisition time for acquiring one target sample image can be obtained from Table 1 as t6.

[0156] Based on the above processing, a first correspondence between data volume and acquisition time can be established. Subsequently, based on this first correspondence, the estimated acquisition time for acquiring a single target sample image can be determined. Then, based on the estimated acquisition time for a single target sample image and the number of sample images, the total acquisition time for acquiring all target sample images can be determined. The acquisition time will vary depending on the number of sample images. Compared to directly determining the total acquisition time for each target sample image, this method improves the accuracy of the determined total acquisition time.

[0157] In some embodiments, Figure 3 Based on this, see Figure 4 After step S105, the method may further include the following steps:

[0158] S106: After the target training task is completed, calculate the ratio of the total acquisition time of each target sample image to the number of each target sample image during the execution of the target training task, and obtain the average acquisition time of each target sample image.

[0159] S107: Update the first correspondence based on the average data volume and average acquisition time of each target sample image.

[0160] After the target training task is completed, the training server can obtain the total acquisition time of each target sample image during the execution of the target training task, and calculate the ratio of this total acquisition time to the number of each target sample image to obtain the average acquisition time of each target sample image. This average acquisition time is the acquisition time for acquiring one target sample image. The training server determines the acquisition time corresponding to the average data volume of each target sample image in the first relation, and calculates the mean of the determined acquisition time and the average acquisition time of each target sample image to obtain the updated average acquisition time. Furthermore, the correspondence between this average data volume and the updated average acquisition time is recorded to obtain the updated first correspondence.

[0161] Based on the above processing, the first correspondence can be updated according to the acquisition time of each target sample image during the execution of the target training task. As the number of training tasks executed by the training server increases, the data in the first correspondence will become more complete and accurate. Subsequently, the data preparation time for other training tasks can be determined based on the updated first correspondence, which can improve the accuracy of the determined data preparation time.

[0162] Regarding step S103, in one implementation, the execution time corresponding to the training parameter group in the second correspondence is: the task execution time for executing historical training tasks.

[0163] Based on each historical training task, the training server determines multiple training parameter groups. Each training parameter group includes: algorithm type, image type, preset image width range, preset image height range, and batch size. For each historical training task, the training server obtains the task execution time and determines the training parameter group to which that historical training task belongs. Then, historical training tasks belonging to the same training parameter group are grouped together, resulting in multiple training task groups (i.e., the first training task group).

[0164] The training parameters belonging to the same training parameter group means that: the algorithm type of the image processing model corresponding to each historical training task is the algorithm type of the training parameter group; the image type of each historical sample image is the image type of the training parameter group; the batch size of each historical training task is the batch size of the training parameter group; the image width of each historical sample image belongs to the preset image width range of the training parameter group; and the image height of each historical sample image belongs to the preset image height range of the training parameter group.

[0165] The training server calculates the average execution time of each historical training task in the first training task group, which is used as the execution time of the corresponding training parameter group. Then, the training server records the correspondence between each training parameter group and the task execution time, thus obtaining the second correspondence between training parameter groups and execution times.

[0166] The training server can determine the execution time corresponding to the training parameter group to which the target training parameter belongs from the second correspondence between the preset training parameter group and the execution time, and use it as the task execution time of the target training task.

[0167] In another implementation, the execution time corresponding to the training parameter group in the second correspondence is: the single execution time of one iteration of the historical training task.

[0168] Correspondingly, in Figure 1 Based on this, see Figure 5 Step S103 may include the following steps:

[0169] S1031: In the second correspondence between the preset training parameter group and the execution time, determine the execution time corresponding to the training parameter group to which the target training parameter belongs, and use it as the estimated single execution time for the target training task to perform one iteration calculation.

[0170] S1032: Calculate the total number of iterations for the target training task based on the number of target sample images, the batch size of the target training task, and the batch iteration number of the target training task.

[0171] S1033: Calculate the product of the total number of iterations of the target training task and the estimated single execution time of the target training task to obtain the task execution time of the target training task.

[0172] In some embodiments, the algorithm parameters of the target image processing model are the algorithm type of the target image processing model. Algorithm types include: object detection model, image segmentation model, image classification model, and image recognition model, etc. The image parameters of the target sample image include: image type, width, and height. Since each target sample image is used to train the target image processing model, the image type, width, and height of each target sample image are the same. The image type of the target sample image represents the data format of the target sample image. The execution parameters of the target training task include: the batch size of the target training task. The batch size indicates that a batch_size (batch size) of target sample images are taken for training each time the target training task is executed. The training parameter group in the second correspondence is determined based on each historical training task. One training parameter group in the second correspondence includes: algorithm type, image type, preset image width range, preset image height range, and batch size.

[0173] The target training parameters belong to a training parameter group, which means that the algorithm type in the training parameter group is the same as the algorithm type of the target image processing model, the image type is the same as the image type of the target sample image, the batch size is the same as the batch size of the target training task, the preset image width range includes the image width of the target sample image, and the preset image height range includes the image height of the target sample image.

[0174] After determining the estimated execution time of one iteration of the target training task, the training server calculates the total number of iterations for the target training task. For example, the training server calculates the total number of iterations for the target training task based on the following formula: Total number of iterations for the target training task = (Number of images in the dataset / batch_size) × epoch. Where, the number of images in the dataset is the total number of target sample images; batch_size represents the batch size of the target training task; and epoch represents the number of batch iterations for the target training task. The number of batch iterations indicates the number of times the target training task performs iteration calculations based on each batch of target sample images.

[0175] Furthermore, the training server calculates the execution time of the target training task based on the following formula: Execution time of the target training task = Total number of iterations of the target training task × Training time of a single iteration. Here, the training time of a single iteration represents the estimated execution time of one iteration of the target training task.

[0176] Based on the above processing, the estimated execution time of one iteration of the target training task can be determined. Based on the estimated execution time of the target training task and the total number of iterations, the task execution time of the target training task can be determined. Since the task execution time of the training task will be different when the total number of iterations is different, the accuracy of the determined task execution time can be improved compared with directly determining the task execution time of the target training task.

[0177] In some embodiments, prior to step S103, the method may further include the following steps:

[0178] Step 1: For each historical training task, calculate the ratio of the task execution time to the total number of iterations of the historical training task to obtain the single execution time of one iteration of the historical training task.

[0179] Step 2: Divide the historical training tasks whose training parameters belong to the same training parameter group into a group to obtain multiple first training task groups.

[0180] One set of training parameters includes: algorithm type, image type, preset image width range, preset image height range, and batch size.

[0181] Step 3: For each first training task group, calculate the average execution time of each historical training task in the first training task group to obtain the execution time of the training parameter group corresponding to the first training task group.

[0182] Step 4: Record the correspondence between each training parameter set and the execution time of a single session to obtain the second correspondence between the training parameter set and the execution time.

[0183] For each historical training task, the training server obtains the task execution time and calculates the ratio of the task execution time to the total number of iterations for that historical training task, thus obtaining the single execution time of one iteration for that historical training task. The training server determines the training parameter group to which the historical training task belongs. Then, historical training tasks belonging to the same training parameter group are grouped together, resulting in multiple training task groups (i.e., in the first training task). Next, the training server calculates the average single execution time of each historical training task in the first training task group, which is used as the single execution time corresponding to that training parameter group. Finally, the training server records the correspondence between each training parameter group and the single execution time, obtaining a second correspondence between training parameter groups and execution times.

[0184] For example, see Table 2, which is a table showing the correspondence between training parameter sets and execution durations provided in an embodiment of this application. In Table 2, the average time of one iteration is the execution duration per iteration.

[0185] Table 2

[0186]

[0187] If the algorithm type of the target image processing model is algorithm type A, the image type of the target sample image is image type A, the batch size of the target training task is s1, the image width of the target sample image is between [w1, w2], and the image height of the target sample image is between [h1, h2], then based on Table 2 above, the estimated single execution time of the target training task for one iteration is t1.

[0188] Based on the above processing, a second correspondence between parameter sets and execution durations can be trained. Subsequently, the estimated single execution duration of one iteration of the target training task can be determined based on the second correspondence. Then, based on the estimated single execution duration and the total number of iterations of the target training task, the task execution duration of the target training task can be determined. Since the task execution duration will vary depending on the total number of iterations, this method improves the accuracy of the determined task execution duration compared to directly determining the task execution duration.

[0189] In some embodiments, Figure 5 Based on this, see Figure 6 After step S105, the method may further include the following steps:

[0190] S108: After the target training task is completed, calculate the ratio of the task execution time to the total number of iterations of the target training task to obtain the actual single execution time of the target training task for one iteration.

[0191] S109: Update the second correspondence based on the actual single execution time of one iteration of the target training parameters and the target training task.

[0192] After the target training task is completed, the training server can obtain the task execution time and calculate the ratio of this execution time to the total number of iterations of the target training task, thus obtaining the actual single execution time of one iteration of the target training task. Then, the training server determines the single execution time corresponding to the training parameter group to which the target training parameters belong in the second relationship, and calculates the average of the determined single execution time and the single execution time of one iteration of the target training task, thus obtaining the updated single execution time. Finally, the correspondence between the training parameter group and the updated single execution time is recorded, resulting in the updated second correspondence.

[0193] Based on the above processing, the second correspondence can be updated based on the execution time of the target training task. As the number of training tasks executed by the training server increases, the data in the second correspondence will become more complete and accurate. Subsequently, the execution time of other training tasks can be determined based on the updated second correspondence, which can improve the accuracy of the determined task execution time.

[0194] Regarding step S104, in one implementation, if the training server only needs to execute the target training task, the training server calculates the sum of the data preparation time and the task execution time of the target training task to obtain the total training time of the target training task.

[0195] In another implementation, if the training server needs to execute other training tasks before executing the target training task, the target training task needs to wait for a period of time before it can be executed. In this case, the training server can also calculate the waiting time of the target training task and calculate the total training time of the target training task based on the waiting time of the target training task.

[0196] Correspondingly, in Figure 1 Based on this, see Figure 7 Before step S104, the method may further include the following steps:

[0197] S110: Based on a preset task scheduling strategy, determine the graphics card in the training server used to execute the target training task, and designate it as the target graphics card.

[0198] S111: Determine the training task that the target graphics card needs to perform before executing the target training task, and designate it as the first training task.

[0199] S112: From the first training task, determine the first training task currently being executed by the target graphics card as the second training task, and determine the first training task to be executed by the target graphics card as the third training task.

[0200] S113: Calculate the sum of the remaining execution time of the second training task, the data preparation time of the third training task, and the task execution time to obtain the waiting time of the target training task.

[0201] Accordingly, step S104 may include the following steps:

[0202] S1041: Calculate the sum of the waiting time, data preparation time, and task execution time of the target training task to obtain the total training time of the target training task.

[0203] A training server can include multiple graphics cards, also known as GPUs (Graphics Processing Units). Each graphics card can only execute one training task at a time. When the training server receives a target training task, it can determine the graphics card (i.e., the target graphics card) to execute the target training task based on a preset task scheduling strategy.

[0204] The task scheduling strategy can be as follows: Training tasks are assigned to the respective graphics cards in the order they are acquired. For example, if the training server has two graphics cards, the server might assign the first training task to graphics card 1, the second to graphics card 2, the third to graphics card 1, the fourth to graphics card 2, and so on.

[0205] Accordingly, once the target graphics card is identified, the first training task that the target graphics card needs to perform before executing the target training task can be obtained. From the first training task, the training server can determine the second training task currently being performed by the target graphics card, as well as the third training task to be performed by the target graphics card.

[0206] Since the target training task can only begin after the second and third training tasks have been completed, the training server can calculate the remaining execution time of the second training task, the data preparation time of the third training task, and the sum of the task execution time based on the following formula, and thus obtain the waiting time of the target training task.

[0207] Queueing time = Remaining training time for "currently training" tasks + Total estimated training time for queued training tasks.

[0208] The queuing time is the waiting time for the target training task; the training task that is "training" is the second training task; the remaining time of the training task that is "training" is the remaining execution time of the second training task; the total estimated training time of the training tasks in the queue is the sum of the data preparation time and the task execution time of each third training task.

[0209] For the third training task that has not yet started training, the training server estimates and determines the data preparation time and task execution time of the third training task based on the algorithm type of the image processing model corresponding to the third training task, the data volume and number of the corresponding second sample images, and the execution parameters of the third training task. The method for determining the data preparation time and task execution time of the third training task is similar to the method for determining the data preparation time and task execution time of the target training task, and can be referred to the relevant description in the foregoing embodiments.

[0210] For the user, the total training time of the entire target training task is the time from when the user submits the target training task to when the target image processing model completes training. The total training time includes the queuing time, data preparation time, and task execution time of the target training task. Then, the training server calculates the sum of the waiting time, data preparation time, and task execution time of the target training task to obtain the total training time of the target training task.

[0211] In some embodiments, for a second training task that has already started training, the training server can calculate and determine the remaining execution time of the second training task based on the actual execution status of the second training task, and the remaining execution time of the second training task can be determined with relatively high accuracy.

[0212] Correspondingly, in Figure 7 Based on this, see Figure 8 Before step S113, the method may further include the following steps:

[0213] S114: Calculate the total number of iterations for the second training task based on the number of first sample images corresponding to the second training task, the batch size of the second training task, and the batch iteration number of the second training task.

[0214] S115: Calculate the difference between the total number of iterations for the second training task and the number of iterations calculated from the iterations already performed in the second training task, to obtain the remaining number of iterations for the second training task.

[0215] S116: Calculate the remaining execution time of the second training task based on the single execution time of the iterations already executed for the second training task and the remaining number of iterations for the second training task.

[0216] The training server calculates the total number of iterations for the second training task in a similar way to calculate the total number of iterations for the target training task, as described in the foregoing embodiments.

[0217] During the execution of the second training task, the training server records the duration of each iteration after each calculation and determines the duration of any unexecuted iterations based on the durations of the already executed iterations. For example, the training server can obtain the duration of the most recent iteration as the duration of any unexecuted iteration; or, the training server can calculate the average duration of the already executed iterations as the duration of any unexecuted iteration.

[0218] Then, the training server calculates the product of the remaining number of iterations for the second training task and the duration of a single iteration for the unexecuted iterations to obtain the remaining execution time of the second training task.

[0219] For example, the training server can calculate the remaining execution time of the second training task based on the following formula: Remaining training time = (Total number of iterations of the second training task - Number of iterations already performed) × Training time of a single iteration. Wherein, remaining training time is the remaining execution time of the second training task; number of iterations already performed is the number of iterations already performed in the second training task; and training time of a single iteration is the duration of a single iteration of the unperformed iterations in the second training task.

[0220] In step S105, after determining the total training time of the target training task, the training server can send the determined total training time to the terminal. Correspondingly, the terminal can display the total training time of the target training task for the user to view, allowing the user to know exactly when the target image processing model will complete training, thus improving the user experience.

[0221] For example, see Figure 9 , Figure 9 This is a schematic diagram of a terminal display interface provided in an embodiment of this application.

[0222] Figure 9 The name of the display interface shown is "AI Training Platform". Figure 9 The displayed interface provides search criteria selection buttons, including: algorithm type, training status, and creation date. The interface also provides a search input box, namely the "Please enter model name" input box. Users can... Figure 9Enter the search criteria for the training task and the name of the image processing model, and click the "Search" button to input the search command into the terminal. The terminal will then display training tasks that match the user's search criteria. Alternatively, the user can click the "Reset" button to re-enter the search criteria. The "Model Center" is used to manage submitted image processing model training tasks and to validate trained image processing models. "Model Management" is used to manage currently executing training tasks.

[0223] Figure 9 The displayed interface also shows task information for the training tasks of various types of image processing models. For example, for the training task of the "instance segmentation" type image processing model, the creator of this training task is A, and the creation time is T1; for the training task of the "object detection" type image processing model, the creator of this training task is B, and the creation time is T2; for the training task of the "image segmentation" type image processing model, the creator of this training task is C, and the creation time is T3; for the training task of the "image classification" type image processing model, the creator of this training task is E, and the creation time is T5. The display interface also shows operation buttons for each training task, such as "Edit" and "Delete" buttons.

[0224] Figure 9 The display interface can also show information such as the training status, training start time, estimated remaining time, and total training time for training tasks of various types of image processing models. The estimated remaining time is calculated as: Total training time of the training task - (Current time - Submission time of the training task). The total training time of the training task is calculated as: Queuing time + Data preparation time + Task execution training time. The total training time is the total training time determined based on the model training time determination method provided in this application embodiment.

[0225] For example, considering three training tasks for an image processing model of the "image segmentation" type, the training status of the first task is "training in progress," the trainer is C, the training start time is t11, the estimated remaining time is t12, and the total training time is t13. The training status of the second task is "waiting in line," the trainer is C, the training start time is t21, the estimated remaining time is t22, and the total training time is t23. The training status of the third task is "training successful," the trainer is C, the training start time is t31, the estimated remaining time is t32, and the total training time is t33.

[0226] Figure 9The display interface also shows "Operation" buttons for training tasks of various types of image processing models. Among them, the "Stop Training" button is used to instruct the terminal to stop the training task; the "Details" button is used to instruct the terminal to display detailed information about the training task; the "Download" button is used to instruct the terminal to download the trained image processing model; the "Delete" button is used to instruct the terminal to delete the training task; and the "Verify" button is used to instruct the terminal to verify the trained image processing model.

[0227] and Figure 1 For the corresponding method implementation examples, see [link to relevant documentation]. Figure 10 , Figure 10 This is a structural diagram of a model training duration determination device provided in an embodiment of this application. The device is applied to a training server in a model training platform, which also includes a terminal. The device includes:

[0228] The target training task acquisition module 1001 is used to acquire the target training task; wherein, the target training task is used to train the target image processing model based on the target sample image;

[0229] The data preparation time determination module 1002 is used to determine the acquisition time required to acquire the target sample image based on the data volume of the target sample image and a preset first correspondence between the data volume and the acquisition time, thereby obtaining the data preparation time of the target training task; wherein, the first correspondence is determined based on the data volume of historical sample images corresponding to historical training tasks already executed by the training server and the acquisition time of acquiring the historical sample images;

[0230] The task execution duration determination module 1003 is used to determine the task execution duration of the target training task based on the target training parameters of the target training task and a second correspondence between a preset training parameter set and the execution duration; wherein, the target training parameters include: the algorithm parameters of the target image processing model, the image parameters of the target sample image, and the execution parameters of the target training task; the second correspondence is determined based on the training parameters and task execution duration of the historical training tasks;

[0231] The total training time determination module 1004 is used to calculate the total training time of the target training task based on the data preparation time and task execution time of the target training task.

[0232] The total training time display module 1005 is used to send the total training time of the target training task to the terminal so that the terminal can display the total training time of the target training task.

[0233] Optionally, the device further includes:

[0234] The target graphics card determination module is used to determine the graphics card in the training server used to execute the target training task as the target graphics card before the total training time determination module 1004 executes the data preparation time and task execution time based on the target training task and calculates the total training time of the target training task.

[0235] The first training task determination module is used to determine the training task that the target graphics card needs to perform before executing the target training task, as the first training task;

[0236] The second training task determination module is used to determine, from the first training task, the first training task currently being executed by the target graphics card as the second training task, and to determine the first training task to be executed by the target graphics card as the third training task.

[0237] The waiting time determination module is used to calculate the sum of the remaining execution time of the second training task, the data preparation time of the third training task, and the task execution time to obtain the waiting time of the target training task;

[0238] The total training time determination module 1004 is specifically used to calculate the sum of the waiting time, data preparation time and task execution time of the target training task to obtain the total training time of the target training task.

[0239] Optionally, the device further includes:

[0240] The total iteration count determination module is used to calculate the total number of iterations of the second training task before the waiting time determination module calculates the remaining execution time of the second training task, the data preparation time of the third training task, and the sum of the task execution time to obtain the waiting time of the target training task. This is done based on the number of first sample images corresponding to the second training task, the batch size of the second training task, and the batch iteration count of the second training task.

[0241] The remaining iteration count determination module is used to calculate the difference between the total number of iterations of the second training task and the number of iterations calculated by the iterations already performed in the second training task, so as to obtain the remaining number of iterations of the second training task;

[0242] The remaining execution time determination module is used to calculate the remaining execution time of the second training task based on the single execution time calculated from the iterations already executed by the second training task and the remaining number of iterations of the second training task.

[0243] Optionally, the task execution time determination module 1003 is specifically used to determine the execution time corresponding to the training parameter group to which the target training parameter belongs in the second correspondence between the preset training parameter group and the execution time, as the estimated single execution time of the target training task for one iteration calculation;

[0244] The total number of iterations for the target training task is calculated based on the number of target sample images, the batch size of the target training task, and the batch iteration number of the target training task.

[0245] The execution time of the target training task is obtained by multiplying the total number of iterations of the target training task by the estimated single execution time of the target training task.

[0246] Optionally, the algorithm parameters of the target image processing model include: the algorithm type of the target image processing model; the image parameters of the target sample image include: the image type, width, and height of the target sample image; and the execution parameters of the target training task include: the batch size of the target training task.

[0247] One of the training parameter groups in the second correspondence includes: algorithm type, image type, preset image width range, preset image height range, and batch size;

[0248] The target training parameters belong to a training parameter group, which means that the algorithm type in the training parameter group is the same as the algorithm type of the target image processing model, the image type is the same as the image type of the target sample image, the batch size is the same as the batch size of the target training task, the preset image width range includes the image width of the target sample image, and the preset image height range includes the image height of the target sample image.

[0249] Optionally, the device further includes:

[0250] The first single execution duration determination module is used to determine the task execution duration of the target training task based on the target training parameters of the target training task and the second correspondence between the preset training parameter group and the execution duration before the task execution duration determination module 1003 executes the task execution duration of the target training task. Before that, for each historical training task, the module calculates the ratio of the task execution duration of the historical training task to the total number of iterations of the historical training task to obtain the single execution duration of the historical training task for one iteration calculation.

[0251] The first training task grouping module is used to group historical training tasks whose training parameters belong to the same training parameter group into a group, resulting in multiple first training task groups; wherein, a training parameter group includes: algorithm type, image type, preset image width range, preset image height range, and batch size;

[0252] The second single execution duration determination module is used to calculate the average single execution duration of each historical training task in each first training task group, so as to obtain the single execution duration corresponding to the training parameter group corresponding to the first training task group.

[0253] The second correspondence determination module is used to record the correspondence between each training parameter group and the single execution duration, and obtain the second correspondence between the training parameter group and the execution duration.

[0254] Optionally, the device further includes:

[0255] The actual single execution time determination module is used to calculate the total training time of the target training task after the total training time determination module 1004 calculates the data preparation time and task execution time based on the target training task, and after the target training task is completed, calculate the ratio of the task execution time of the target training task to the total number of iterations of the target training task, and obtain the actual single execution time of the target training task for one iteration calculation.

[0256] The second correspondence update module is used to update the second correspondence based on the target training parameters and the actual execution time of one iteration of the target training task.

[0257] Optionally, the data preparation time determination module 1002 is specifically used to calculate the ratio of the total data volume to the number of each target sample image, and obtain the average data volume of each target sample image.

[0258] In the first correspondence between the preset data volume and the acquisition time, the acquisition time corresponding to the data volume with the smallest difference from the calculated average data volume is determined, and the estimated acquisition time for acquiring a target sample image is obtained.

[0259] Calculate the product of the estimated acquisition time of a target sample image and the number of target sample images to obtain the total acquisition time required to acquire each target sample image, which is used as the data preparation time for the target training task.

[0260] Optionally, the device further includes:

[0261] The average data volume determination module is used to determine the acquisition time required to acquire the target sample image based on the data volume of the target sample image and the first correspondence between the preset data volume and the acquisition time in the data preparation time determination module 1002, and before obtaining the data preparation time of the target training task, it performs the following: for each historical training task, it calculates the ratio of the total data volume to the number of historical sample images corresponding to the historical training task, and obtains the average data volume of each historical sample image.

[0262] The first average acquisition time determination module is used to calculate the ratio of the total acquisition time of each historical sample image to the number of each historical sample image when performing the historical training task, so as to obtain the average acquisition time of each historical sample image.

[0263] The second training task grouping module is used to divide historical training tasks with the same average data volume of each historical sample image into a group, resulting in multiple second training task groups.

[0264] The second average acquisition time determination module is used to calculate the mean of the average acquisition time of each historical training task in the second training task group for each second training task group, so as to obtain the average acquisition time corresponding to the average data volume of the second training task group.

[0265] The first correspondence determination module is used to record the correspondence between each average data volume and the average acquisition time, and to obtain the first correspondence between data volume and acquisition time.

[0266] Optionally, the device includes:

[0267] The third average acquisition time determination module is used to calculate the average acquisition time of each target sample image after the total training time determination module 1004 performs the data preparation time and task execution time based on the target training task and calculates the total training time of the target training task. After the target training task is completed, the module calculates the ratio of the total acquisition time of each target sample image to the number of each target sample image during the execution of the target training task, and obtains the average acquisition time of each target sample image.

[0268] The first correspondence update module is used to update the first correspondence based on the average data volume and average acquisition time of each target sample image.

[0269] The model training duration determination device provided in this application embodiment can determine the data preparation duration of the target training task based on the data volume of the target sample image and a preset first correspondence between the data volume and the acquisition duration, and determine the task execution duration of the target training task based on the target training parameters of the target training task and a preset second correspondence between the training parameters and the execution duration. Furthermore, based on the data preparation duration and task execution duration of the target training task, the total training duration of the target training task is calculated and displayed to the user, allowing the user to know exactly when the target image processing model will complete training, thus improving the user experience.

[0270] This application also provides a model training platform, which includes a training server and a terminal; wherein...

[0271] The training server is used to acquire a target training task; wherein the target training task is used to train a target image processing model based on target sample images; based on the data volume of the target sample images and a preset first correspondence between data volume and acquisition time, the acquisition time required to acquire the target sample images is determined, thus obtaining the data preparation time of the target training task; wherein the first correspondence is determined based on the data volume of historical sample images corresponding to historical training tasks executed by the training server and the acquisition time of acquiring the historical sample images; based on the target training parameters of the target training task and a preset second correspondence between training parameter groups and execution time, the task execution time of the target training task is determined; wherein the target training parameters include: algorithm parameters of the target image processing model, image parameters of the target sample images, and execution parameters of the target training task; the second correspondence is determined based on the training parameters and task execution time of the historical training tasks; based on the data preparation time and task execution time of the target training task, the total training time of the target training task is calculated; and the total training time of the target training task is sent to the terminal.

[0272] The terminal is used to display the total training time of the received target training task.

[0273] Based on the model training platform provided in this application embodiment, the data preparation time for the target training task can be determined based on the data volume of the target sample image and a preset first correspondence between the data volume and the acquisition time. Furthermore, based on the target training parameters of the target training task and a preset second correspondence between the training parameters and the execution time, the task execution time of the target training task can be determined. Then, based on the data preparation time and task execution time of the target training task, the total training time of the target training task is calculated and displayed to the user, allowing the user to know exactly when the target image processing model will be trained, thus improving the user experience.

[0274] This application also provides a training server, such as... Figure 11 As shown, it includes:

[0275] Memory 1101 is used to store computer programs;

[0276] When processor 1102 executes the program stored in memory 1101, it performs the following steps:

[0277] Obtain the target training task; wherein, the target training task is used to train the target image processing model based on the target sample image;

[0278] Based on the data volume of the target sample image and a preset first correspondence between the data volume and the acquisition time, the acquisition time required to acquire the target sample image is determined, and the data preparation time for the target training task is obtained; wherein, the first correspondence is determined based on the data volume of historical sample images corresponding to historical training tasks executed by the training server and the acquisition time for acquiring the historical sample images;

[0279] Based on the target training parameters of the target training task and the second correspondence between the preset training parameter set and the execution time, the task execution time of the target training task is determined; wherein, the target training parameters include: the algorithm parameters of the target image processing model, the image parameters of the target sample image, and the execution parameters of the target training task; the second correspondence is determined based on the training parameters and task execution time of the historical training tasks;

[0280] Based on the data preparation time and task execution time of the target training task, calculate the total training time of the target training task;

[0281] The total training time of the target training task is sent to the terminal in the model training platform so that the terminal displays the total training time of the target training task.

[0282] Furthermore, the aforementioned training server may also include a communication bus and / or a communication interface, and the processor 1102, the communication interface, and the memory 1101 communicate with each other through the communication bus.

[0283] The communication bus mentioned in the training server above can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used to represent it in the diagram, but this does not mean that there is only one bus or one type of bus.

[0284] The communication interface is used for communication between the training server and other devices.

[0285] The memory may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0286] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0287] In another embodiment provided in this application, a computer-readable storage medium is also provided, which stores a computer program that, when executed by a processor, implements the steps of any of the above-described methods for determining the training duration of a model.

[0288] In another embodiment provided in this application, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to execute any of the model training duration determination methods in the above embodiments.

[0289] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or other storage media (e.g., solid state disk (SSD)).

[0290] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0291] The various embodiments in this specification are described in a related manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the embodiments of apparatus, training servers, model training platforms, computer-readable storage media, and computer program products are basically similar to the method embodiments, and therefore the descriptions are relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0292] The above description is merely a preferred embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application are included within the scope of protection of this application.

Claims

1. A method for determining model training duration, characterized in that, The method is applied to a training server in a model training platform, the model training platform further including a terminal, and the method includes: Obtain the target training task; wherein, the target training task is used to train the target image processing model based on the target sample image; Based on the data volume of the target sample image and a preset first correspondence between the data volume and the acquisition time, the acquisition time required to acquire the target sample image is determined, and the data preparation time for the target training task is obtained; wherein, the first correspondence is determined based on the data volume of historical sample images corresponding to historical training tasks executed by the training server and the acquisition time for acquiring the historical sample images; Based on the target training parameters of the target training task and the second correspondence between the preset training parameter set and the execution time, the task execution time of the target training task is determined; wherein, the target training parameters include: the algorithm parameters of the target image processing model, the image parameters of the target sample image, and the execution parameters of the target training task; the second correspondence is determined based on the training parameters and task execution time of the historical training tasks; Based on the data preparation time and task execution time of the target training task, calculate the total training time of the target training task; The total training time of the target training task is sent to the terminal so that the terminal displays the total training time of the target training task; The determination of the task execution time of the target training task based on the target training parameters of the target training task and the second correspondence between the preset training parameter set and the execution time includes: In the second correspondence between the preset training parameter group and the execution time, the execution time corresponding to the training parameter group to which the target training parameter belongs is determined as the estimated single execution time of the target training task for one iteration calculation; The total number of iterations for the target training task is calculated based on the number of target sample images, the batch size of the target training task, and the batch iteration number of the target training task. The execution time of the target training task is obtained by multiplying the total number of iterations of the target training task by the estimated single execution time of the target training task.

2. The method according to claim 1, characterized in that, Before calculating the total training time of the target training task based on the data preparation time and task execution time of the target training task, the method further includes: Based on a preset task scheduling strategy, the graphics card in the training server used to execute the target training task is determined as the target graphics card; The training task that the target graphics card needs to perform before executing the target training task is determined and designated as the first training task; From the first training task, determine the first training task currently being executed by the target graphics card as the second training task, and determine the first training task to be executed by the target graphics card as the third training task; The remaining execution time of the second training task, the data preparation time of the third training task, and the sum of the task execution time are calculated to obtain the waiting time of the target training task. The calculation of the total training time for the target training task based on the data preparation time and task execution time includes: The total training time of the target training task is obtained by summing the waiting time, data preparation time, and task execution time of the target training task.

3. The method according to claim 2, characterized in that, Before calculating the sum of the remaining execution time of the second training task, the data preparation time of the third training task, and the task execution time to obtain the waiting time of the target training task, the method further includes: The total number of iterations for the second training task is calculated based on the number of first sample images corresponding to the second training task, the batch size of the second training task, and the batch iteration number of the second training task. The difference between the total number of iterations for the second training task and the number of iterations already performed in the second training task is calculated to obtain the remaining number of iterations for the second training task. The remaining execution time of the second training task is calculated based on the single execution time of the iterations already performed in the second training task and the remaining number of iterations of the second training task.

4. The method according to claim 1, characterized in that, The algorithm parameters of the target image processing model are: the algorithm type of the target image processing model; The image parameters of the target sample image include: the image type, width, and height of the target sample image; the execution parameters of the target training task include: the batch size of the target training task; One of the training parameter groups in the second correspondence includes: algorithm type, image type, preset image width range, preset image height range, and batch size; The target training parameters belong to a training parameter group, which means that the algorithm type in the training parameter group is the same as the algorithm type of the target image processing model, the image type is the same as the image type of the target sample image, the batch size is the same as the batch size of the target training task, the preset image width range includes the image width of the target sample image, and the preset image height range includes the image height of the target sample image.

5. The method according to claim 1, characterized in that, Before determining the execution time of the target training task based on the target training parameters of the target training task and the second correspondence between the preset training parameter set and the execution time, the method further includes: For each historical training task, the ratio of the task execution time to the total number of iterations of the historical training task is calculated to obtain the single execution time of one iteration of the historical training task. Historical training tasks whose training parameters belong to the same training parameter group are grouped together to obtain multiple first training task groups; each training parameter group includes: algorithm type, image type, preset image width range, preset image height range, and batch size; For each first training task group, calculate the average execution time of each historical training task in the first training task group to obtain the execution time of the training parameter group corresponding to the first training task group. Record the correspondence between each training parameter set and the execution duration to obtain the second correspondence between the training parameter set and the execution duration.

6. The method according to claim 1, characterized in that, After calculating the total training time of the target training task based on the data preparation time and task execution time of the target training task, the method further includes: After the target training task is completed, the ratio of the task execution time to the total number of iterations of the target training task is calculated to obtain the actual single execution time of the target training task for one iteration. The second correspondence is updated based on the target training parameters and the actual execution time of one iteration of the target training task.

7. The method according to claim 1, characterized in that, The determination of the required acquisition time for acquiring the target sample image based on the data volume of the target sample image and a preset first correspondence between the data volume and the acquisition time, thereby obtaining the data preparation time for the target training task, includes: Calculate the ratio of the total data volume to the number of target sample images to obtain the average data volume of each target sample image; In the first correspondence between the preset data volume and the acquisition time, the acquisition time corresponding to the data volume with the smallest difference from the calculated average data volume is determined, and the estimated acquisition time for acquiring a target sample image is obtained. Calculate the product of the estimated acquisition time of a target sample image and the number of target sample images to obtain the total acquisition time required to acquire each target sample image, which is used as the data preparation time for the target training task.

8. The method according to claim 7, characterized in that, Before determining the acquisition time required to acquire the target sample image based on the data volume of the target sample image and a preset first correspondence between the data volume and the acquisition time, and obtaining the data preparation time for the target training task, the method further includes: For each historical training task, the ratio of the total data volume to the number of historical sample images corresponding to that historical training task is calculated to obtain the average data volume of each historical sample image. The average acquisition time of each historical sample image is obtained by calculating the ratio of the total acquisition time to the number of historical sample images when performing the historical training task. Historical training tasks with the same average data volume for each historical sample image are grouped together to obtain multiple second training task groups. For each second training task group, calculate the mean of the average acquisition time corresponding to each historical training task in the second training task group, and obtain the average acquisition time corresponding to the average data volume of the second training task group. Record the correspondence between the average data volume and the average acquisition time to obtain the first correspondence between data volume and acquisition time.

9. The method according to claim 7, characterized in that, After calculating the total training time of the target training task based on the data preparation time and task execution time of the target training task, the method further includes: After the target training task is completed, the ratio of the total acquisition time of each target sample image to the number of each target sample image is calculated to obtain the average acquisition time of each target sample image. The first correspondence is updated based on the average data volume and average acquisition time of each target sample image.

10. A model training platform, characterized in that, The model training platform includes: a training server and a terminal; wherein... The training server is used to acquire a target training task; wherein the target training task is used to train a target image processing model based on target sample images; based on the data volume of the target sample images and a preset first correspondence between data volume and acquisition time, the acquisition time required to acquire the target sample images is determined, thus obtaining the data preparation time of the target training task; wherein the first correspondence is determined based on the data volume of historical sample images corresponding to historical training tasks executed by the training server and the acquisition time of acquiring the historical sample images; based on the target training parameters of the target training task and a preset second correspondence between training parameter groups and execution time, the task execution time of the target training task is determined; wherein the target training parameters include: algorithm parameters of the target image processing model, image parameters of the target sample images, and execution parameters of the target training task; the second correspondence is determined based on the training parameters and task execution time of the historical training tasks; based on the data preparation time and task execution time of the target training task, the total training time of the target training task is calculated; and the total training time of the target training task is sent to the terminal. The terminal is used to display the total training time of the target training task received. The training server is specifically used for: In the second correspondence between the preset training parameter group and the execution time, the execution time corresponding to the training parameter group to which the target training parameter belongs is determined as the estimated single execution time of the target training task for one iteration calculation; The total number of iterations for the target training task is calculated based on the number of target sample images, the batch size of the target training task, and the batch iteration number of the target training task. The execution time of the target training task is obtained by multiplying the total number of iterations of the target training task by the estimated single execution time of the target training task.

11. A device for determining model training duration, characterized in that, The device is used in a training server of a model training platform, the model training platform further includes a terminal, and the device includes: The target training task acquisition module is used to acquire the target training task; wherein, the target training task is used to train the target image processing model based on the target sample image; The data preparation time determination module is used to determine the acquisition time required to acquire the target sample image based on the data volume of the target sample image and a preset first correspondence between the data volume and the acquisition time, thereby obtaining the data preparation time of the target training task; wherein, the first correspondence is determined based on the data volume of historical sample images corresponding to historical training tasks already executed by the training server and the acquisition time of acquiring the historical sample images; The task execution duration determination module is used to determine the task execution duration of the target training task based on the target training parameters of the target training task and a second correspondence between a preset training parameter set and the execution duration; wherein, the target training parameters include: the algorithm parameters of the target image processing model, the image parameters of the target sample image, and the execution parameters of the target training task; the second correspondence is determined based on the training parameters and task execution duration of the historical training tasks; The total training time determination module is used to calculate the total training time of the target training task based on the data preparation time and task execution time of the target training task; The total training time display module is used to send the total training time of the target training task to the terminal so that the terminal can display the total training time of the target training task; The task execution time determination module is specifically used to determine the execution time corresponding to the training parameter group to which the target training parameter belongs in the second correspondence between the preset training parameter group and the execution time, as the estimated single execution time of the target training task for one iteration calculation; The total number of iterations for the target training task is calculated based on the number of target sample images, the batch size of the target training task, and the batch iteration number of the target training task. The execution time of the target training task is obtained by multiplying the total number of iterations of the target training task by the estimated single execution time of the target training task.

12. The apparatus according to claim 11, characterized in that, The device further includes: The target graphics card determination module is used to determine the graphics card in the training server used to execute the target training task as the target graphics card before the total training time determination module executes the data preparation time and task execution time based on the target training task and calculates the total training time of the target training task. The first training task determination module is used to determine the training task that the target graphics card needs to perform before executing the target training task, as the first training task; The second training task determination module is used to determine, from the first training task, the first training task currently being executed by the target graphics card as the second training task, and to determine the first training task to be executed by the target graphics card as the third training task. The waiting time determination module is used to calculate the sum of the remaining execution time of the second training task, the data preparation time of the third training task, and the task execution time to obtain the waiting time of the target training task; The total training time determination module is specifically used to calculate the sum of the waiting time, data preparation time and task execution time of the target training task to obtain the total training time of the target training task. The device further includes: The total iteration count determination module is used to calculate the total number of iterations of the second training task before the waiting time determination module calculates the remaining execution time of the second training task, the data preparation time of the third training task, and the sum of the task execution time to obtain the waiting time of the target training task. This is done based on the number of first sample images corresponding to the second training task, the batch size of the second training task, and the batch iteration count of the second training task. The remaining iteration count determination module is used to calculate the difference between the total number of iterations of the second training task and the number of iterations calculated by the iterations already performed in the second training task, so as to obtain the remaining number of iterations of the second training task; The remaining execution time determination module is used to calculate the remaining execution time of the second training task based on the single execution time calculated from the iterations already executed by the second training task and the remaining number of iterations of the second training task; The algorithm parameters of the target image processing model are: the algorithm type of the target image processing model; the image parameters of the target sample image include: the image type, width, and height of the target sample image; the execution parameters of the target training task include: the batch size of the target training task. One of the training parameter groups in the second correspondence includes: algorithm type, image type, preset image width range, preset image height range, and batch size; The target training parameters belong to a training parameter group, which means that the algorithm type in the training parameter group is the same as the algorithm type of the target image processing model, the image type is the same as the image type of the target sample image, the batch size is the same as the batch size of the target training task, the preset image width range includes the image width of the target sample image, and the preset image height range includes the image height of the target sample image. The device further includes: The first single execution duration determination module is used to determine the task execution duration of the target training task based on the target training parameters of the target training task and the second correspondence between the preset training parameter group and the execution duration before the task execution duration determination module executes the task execution duration of the target training task. For each historical training task, the module calculates the ratio of the task execution duration of the historical training task to the total number of iterations of the historical training task to obtain the single execution duration of the historical training task for one iteration calculation. The first training task grouping module is used to group historical training tasks whose training parameters belong to the same training parameter group into a group, resulting in multiple first training task groups; wherein, a training parameter group includes: algorithm type, image type, preset image width range, preset image height range, and batch size; The second single execution duration determination module is used to calculate the average single execution duration of each historical training task in each first training task group, so as to obtain the single execution duration corresponding to the training parameter group corresponding to the first training task group. The second correspondence determination module is used to record the correspondence between each training parameter group and the single execution duration, and to obtain the second correspondence between the training parameter group and the execution duration. The device further includes: The actual single execution time determination module is used to calculate the total training time of the target training task based on the data preparation time and task execution time of the target training task after the total training time determination module has executed the target training task. After the target training task has been executed, the module calculates the ratio of the task execution time of the target training task to the total number of iterations of the target training task to obtain the actual single execution time of the target training task for one iteration. The second correspondence update module is used to update the second correspondence based on the target training parameters and the actual single execution time of one iteration of the target training task. The data preparation time determination module is specifically used to calculate the ratio of the total data volume to the number of target sample images, and to obtain the average data volume of each target sample image. In the first correspondence between the preset data volume and the acquisition time, the acquisition time corresponding to the data volume with the smallest difference from the calculated average data volume is determined, and the estimated acquisition time for acquiring a target sample image is obtained. Calculate the product of the estimated acquisition time of a target sample image and the number of target sample images to obtain the total acquisition time required to acquire each target sample image, which is used as the data preparation time for the target training task. The device further includes: The average data volume determination module is used to determine the acquisition time required to acquire the target sample image based on the data volume of the target sample image and the first correspondence between the preset data volume and the acquisition time in the data preparation time determination module, and to obtain the data preparation time of the target training task before the data preparation time determination module executes the following: for each historical training task, the module calculates the ratio of the total data volume to the number of historical sample images corresponding to the historical training task, and obtains the average data volume of each historical sample image. The first average acquisition time determination module is used to calculate the ratio of the total acquisition time of each historical sample image to the number of each historical sample image when performing the historical training task, so as to obtain the average acquisition time of each historical sample image. The second training task grouping module is used to divide historical training tasks with the same average data volume of each historical sample image into a group, resulting in multiple second training task groups. The second average acquisition time determination module is used to calculate the mean of the average acquisition time of each historical training task in the second training task group for each second training task group, so as to obtain the average acquisition time corresponding to the average data volume of the second training task group. The first correspondence determination module is used to record the correspondence between each average data volume and average acquisition time, and to obtain the first correspondence between data volume and acquisition time. The device includes: The third average acquisition time determination module is used to calculate the average acquisition time of each target sample image after the total training time determination module performs the data preparation time and task execution time based on the target training task and calculates the total training time of the target training task. After the target training task is completed, the module calculates the ratio of the total acquisition time of each target sample image to the number of each target sample image during the execution of the target training task, and obtains the average acquisition time of each target sample image. The first correspondence update module is used to update the first correspondence based on the average data volume and average acquisition time of each target sample image.

13. A training server, characterized in that, include: Memory, used to store computer programs; A processor, when executing a program stored in memory, implements the method described in any one of claims 1-9.

14. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method described in any one of claims 1-9.

Images