A shared AI model training platform design method and device

Abstract

The present invention relates to the field of artificial intelligence, and specifically provides a shared AI model training platform design method, which has the following steps: S1, the terminal user creates a user in the cloud and declares the organization to which he belongs, and the remaining users can join the organization. The first user to create the organization is the administrator by default. After the user is created, a node access instruction is generated, including a cloud service address and a registration verification key for use when registering the edge node; S2, the edge with GPU computing power resources runs the generated registration instruction to become a sub-node under the organization for use by all users of the organization. After the registration is completed, the cloud can view the node computing power usage and the occupied processes involved; S3, the user uploads the standardized format data that has been marked. All users in the same organization of the data set can view and use it, and users from different organizations cannot view and use it. Compared with the existing technology, the present invention enables the sharing and use of platform training resources while isolating each other's data to a certain extent, ensuring the smooth operation and use of the platform.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence, and specifically provides a method and device for designing a shared AI model training platform. Background Art

[0002] Researchers in the fields of machine learning and deep learning need to continuously conduct model training in the hope of obtaining a more effective model for actual production. However, model training is a complex and chaotic process. How to create an orderly shared training platform is a technical problem that technicians in this field urgently need to solve. Summary of the Invention

[0003] In response to the above-mentioned deficiencies in the prior art, the present invention provides a method for designing a shared AI model training platform with strong practicality.

[0004] A further technical task of the present invention is to provide a design device for a shared AI model training platform that is rationally designed, safe and applicable.

[0005] The technical solution adopted by the present invention to solve its technical problem is:

[0006] A method for designing a shared AI model training platform includes the following steps:

[0007] S1. The end user creates a user in the cloud and declares their organization. Other users can join the organization. The first user to create an organization is the administrator by default. After the user is created, a node access instruction is generated, including the cloud service address and registration verification key for edge node registration.

[0008] S2. The edge with GPU computing resources runs the generated registration instruction and becomes a sub-node under the organization for use by all users of the organization. After the registration is completed, the cloud can view the node computing power usage and the involved occupied processes;

[0009] S3. Users upload labeled data in a standardized format. All users in the same organization can view and use the dataset, but users in different organizations cannot view and use it.

[0010] Furthermore, in step S3, a training task is created by selecting the dataset to be used, checking the node computing power and selecting an idle node, and setting various parameters. Once the task is created, it can be executed, and the results will be automatically uploaded to the cloud after the training task is completed.

[0011] When the task is executed, the cloud service sends the data and parameter settings to the edge node for training. All the result data of the training process is transmitted back to the cloud management center in real time.

[0012] Furthermore, the cloud management center includes user management, data center, task management, log monitoring, node management, resource monitoring and model management;

[0013] The terminal needs to be registered in user management;

[0014] The data center stores the standardized data sets uploaded by the terminal users;

[0015] The task management manages the model training tasks created by the terminal user;

[0016] The log monitoring records the changes in important indicator parameters during model training and the consumption of computing resources during the task training process;

[0017] The node management manages the sub-nodes with computing resources within the same organization;

[0018] The resource monitoring monitors the usage of computing power in the subnodes;

[0019] The model management saves the training results.

[0020] Furthermore, in the user management, the terminal user declares his or her organization when registering. Each organization has an independent resource space, and all computing resources and data between organizations are absolutely isolated. Relevant users in the same organization can see each other's data and share all computing resources in the same organization.

[0021] Furthermore, in the data center, the standardized data set contains labeled image or audio file data, and the data storage is externally connected to a public cloud object storage or cloud disk storage.

[0022] Furthermore, in task management, the created model training tasks include start, pause, end, and view to track the entire training process in real time, complete data set selection, hyperparameter setting, and hardware computing resource selection.

[0023] Furthermore, node management manages sub-nodes with computing resources within the same organization, including generating sub-node registration instructions and monitoring sub-node status.

[0024] Furthermore, resource monitoring monitors the computing power usage in the sub-nodes, including GPU usage, video memory usage, CPU usage, memory usage, and resource usage of each process, to control the computing power usage and arrange training tasks.

[0025] Furthermore, model management saves training results, downloads model results offline or publishes them directly to the cloud service store for other users to use.

[0026] A shared AI model training platform design device, characterized by comprising: at least one memory and at least one processor;

[0027] The at least one memory is configured to store a machine-readable program;

[0028] The at least one processor is used to call the machine-readable program to execute a method for designing a shared AI model training platform.

[0029] Compared with the prior art, the shared AI model training platform design method and device of the present invention have the following outstanding beneficial effects:

[0030] The present invention enables relevant scientific research personnel to share and use platform training resources while isolating their data to a certain extent, thereby ensuring smooth operation and use of the platform. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0032] Attachment Figure 1 It is a framework diagram of the design method of a shared AI model training platform. DETAILED DESCRIPTION

[0033] In order to enable those skilled in the art to better understand the solutions of the present invention, the present invention will be further described in detail below in conjunction with specific embodiments. Obviously, the embodiments described are only some embodiments of the present invention, rather than all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0034] A best embodiment is given below:

[0035] like Figure 1 As shown, a shared AI model training platform design method in this embodiment has the following steps:

[0036] S1. The end user creates a user in the cloud and declares their organization. Other users can join the organization. The first user to create an organization is the administrator by default. After the user is created, a node access instruction is generated, including the cloud service address and registration verification key for edge node registration.

[0037] S2. The edge with GPU computing resources runs the generated registration instruction and becomes a sub-node under the organization for use by all users of the organization. After the registration is completed, the cloud can view the node computing power usage and the involved occupied processes;

[0038] S3. Users upload labeled data in a standardized format. All users in the same organization can view and use the data set, but users in different organizations cannot view and use it. To create a training task, select the data set to be used, view the node computing power and select an idle node, and set various parameters. Once the task is created, it can be executed. After the training task is completed, the results will be automatically uploaded to the cloud.

[0039] When the task is executed, the cloud service sends the data and parameter settings to the edge node for training. All the result data of the training process is transmitted back to the cloud management center in real time.

[0040] Furthermore, the cloud management center includes user management, data center, task management, log monitoring, node management, resource monitoring and model management.

[0041] When registering, end users must declare their organization. Each organization has an independent resource space, and all computing resources and data between organizations are absolutely isolated. Relevant users within the same organization can see each other's uploaded data sets, training tasks, training results, etc., and share all computing resources within the organization.

[0042] The data center is used to store standardized training data sets uploaded by end users, which are usually labeled images or audio files. Data storage can be connected to external media such as public cloud object storage or cloud disk storage.

[0043] Task management is used to manage model training tasks created by end users. It includes sub-modules such as start, pause, end, and view to track the entire training process in real time, and complete dataset selection, hyperparameter setting, hardware computing resource selection, etc.

[0044] Log monitoring records the changes in important indicator parameters during model training and the consumption of computing resources during the task training process;

[0045] Node management is used to manage sub-nodes with computing resources within the same organization, including generating sub-node registration instructions and monitoring sub-node status.

[0046] Resource monitoring is used to monitor the computing power usage in the child nodes, including GPU usage, video memory usage, CPU usage, memory usage, and resource usage of each process, so as to control the computing power usage and arrange training tasks reasonably.

[0047] Model management is used to save training results. The final model results generated vary greatly depending on the deep learning framework selected. The model results can be downloaded offline or directly published to the cloud service store for other users to use.

[0048] The edge service connects a machine with GPU computing power to the model training platform and registers the node with the cloud according to instructions provided by the cloud, declaring the organization to which the machine belongs during registration. The edge server can be a computer on the local area network or a virtual machine with GPU computing resources in the public cloud.

[0049] Based on the above method, a shared AI model training platform design device is characterized by comprising: at least one memory and at least one processor;

[0050] The at least one memory is configured to store a machine-readable program;

[0051] The at least one processor is used to call the machine-readable program to execute a method for designing a shared AI model training platform.

[0052] The above-mentioned specific implementation methods are only specific cases of the present invention. The patent protection scope of the present invention includes but is not limited to the above-mentioned specific implementation methods. Any appropriate changes or substitutions made by ordinary technicians in any of the above-mentioned technical fields that conform to the claims of the shared AI model training platform design method and device of the present invention shall fall within the patent protection scope of the present invention.

[0053] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to these embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the appended claims and their equivalents.

Claims

1. A method for designing a shared AI model training platform, characterized in that: The steps are as follows: S1. The end user creates a user in the cloud and declares their organization. Other users can join the organization. The first user to create an organization is the administrator by default. After the user is created, a node access instruction is generated, including the cloud service address and registration verification key for edge node registration. S2. The edge with GPU computing resources runs the generated registration instruction and becomes a sub-node under the organization for use by all users of the organization. After the registration is completed, the cloud can view the node computing power usage and the involved occupied processes; S3. Users upload labeled data in a standardized format. All users in the same organization can view and use the dataset, but users in different organizations cannot view and use it.

2. A shared AI model training platform design method according to claim 1, characterized in that: In step S3, create a training task, select the dataset to be used, check the node computing power, select an idle node, and set various parameters. After the task is created, it can be executed. After the training task is completed, the results will be automatically uploaded to the cloud. When the task is executed, the cloud service sends the data and parameter settings to the edge node for training. All the result data of the training process is transmitted back to the cloud management center in real time.

3. A shared AI model training platform design method according to claim 2, characterized in that: The cloud management center includes user management, data center, task management, log monitoring, node management, resource monitoring and model management; The terminal needs to be registered in user management; The data center stores the standardized data sets uploaded by the terminal users; The task management manages the model training tasks created by the terminal user; The log monitoring records the changes in important indicator parameters during model training and the consumption of computing resources during the task training process; The node management manages the sub-nodes with computing resources within the same organization; The resource monitoring monitors the usage of computing power in the subnodes; The model management saves the training results.

4. A shared AI model training platform design method according to claim 3, characterized in that: In the user management, the end user declares his or her organization when registering. Each organization has an independent resource space, and all computing resources and data between organizations are absolutely isolated. Relevant users in the same organization can see each other's data and share all computing resources in the same organization.

5. A shared AI model training platform design method according to claim 4, characterized in that: In the data center, the standardized data set contains labeled image or audio file data, and the data storage is connected to public cloud object storage or cloud disk storage.

6. A shared AI model training platform design method according to claim 5, characterized in that: In task management, the created model training task includes start, pause, end, and view to track the entire training process in real time, complete dataset selection, hyperparameter setting, and hardware computing resource selection.

7. A shared AI model training platform design method according to claim 6, characterized in that: Node management manages the sub-nodes with computing resources within the same organization, including generating sub-node registration instructions and monitoring sub-node status.

8. A shared AI model training platform design method according to claim 7, characterized in that: Resource monitoring monitors the computing power usage in the sub-nodes, including GPU usage, video memory usage, CPU usage, memory usage, and resource usage of each process. It controls the computing power usage and arranges training tasks.

9. A shared AI model training platform design method according to claim 8, characterized in that: Model management saves training results, downloads model results offline or publishes them directly to the cloud service store for other users to use.

10. A shared AI model training platform design device, characterized in that: include: at least one memory and at least one processor; The at least one memory is configured to store a machine-readable program; The at least one processor is configured to call the machine-readable program to execute the method according to any one of claims 1 to 9.

Images