A model training method, device, storage medium and computer equipment

Abstract

The present application discloses a model training method, device, storage medium and computer equipment; the present application is related to the field of machine learning of artificial intelligence, and the field of cache reading, and the present application can obtain a data request, the data request includes a training data identifier required for model training; query the target training data corresponding to the training data identifier from the local cache, and the local cache is used for storage; when the target training data corresponding to the training data identifier exists in the local cache, the target training data is loaded into a graphics processor, so that the graphics processor performs model training based on the target training data; when the target training data corresponding to the training data identifier does not exist in the local cache, the target training data is obtained from a data cluster, and the target training data is loaded into the graphics processor, so that the graphics processor performs model training based on the target training data; the present application can effectively improve the efficiency of model training.

Description

Technical Field

[0001] The present application relates to the fields of artificial intelligence and storage, and specifically to a model training method, device, storage medium and computer equipment. Background Art

[0002] With the rapid development of artificial intelligence science, more and more problems can be solved in daily life through network models. Before applying network models to solve problems, training the network model is a key step. Model training usually requires a large amount of training data to conduct multiple rounds of model training. Before each round of training, the existing technology needs to obtain the training data required for that round of training from a remote storage cluster. For example, storage systems commonly used in artificial intelligence training scenarios may include ceph (distributed file storage system), and training data can be read from the ceph remote storage cluster through ceph's ceph fuse (user-mode file system client).

[0003] During the research and practice of the prior art, the inventors of the present application discovered that each round of training requires reading training data from a remote storage cluster. This process is limited by factors such as network conditions and may take a long time to read the training data, resulting in a longer model training process in the prior art. Summary of the invention

[0004] The embodiments of the present application provide a model training method, apparatus, storage medium and computer equipment, which can effectively improve the efficiency of model training.

[0005] The present application embodiment provides a model training method, including:

[0006] Obtaining a data request, wherein the data request includes a training data identifier required for model training;

[0007] Querying the target training data corresponding to the training data identifier from a local cache, wherein the local cache is used for storage;

[0008] When the target training data corresponding to the training data identifier exists in the local cache, the target training data is loaded into the graphics processor, so that the graphics processor performs model training based on the target training data;

[0009] When the target training data corresponding to the training data identifier does not exist in the local cache, the target training data is obtained from the data cluster, and the target training data is loaded into the graphics processor, so that the graphics processor performs model training based on the target training data.

[0010] Accordingly, the present application provides a model training device, comprising:

[0011] An acquisition module, used to acquire a data request, wherein the data request includes a training data identifier required for model training;

[0012] A query module, used to query the target training data corresponding to the training data identifier from a local cache, wherein the local cache is used for storage;

[0013] A loading module, configured to load the target training data into a graphics processor when target training data corresponding to the training data identifier exists in the local cache, so that the graphics processor performs model training based on the target training data;

[0014] The cluster acquisition module is used to acquire the target training data from the data cluster when the target training data corresponding to the training data identifier does not exist in the local cache, and load the target training data into the graphics processor, so that the graphics processor performs model training based on the target training data.

[0015] In some embodiments, the training data identifier includes a first data identifier and a second data identifier, and the loading module is specifically used to:

[0016] When first training data corresponding to the first data identifier exists in the local cache, the first training data is loaded into the graphics processor, and second training data corresponding to the second data identifier obtained from the data cluster is loaded into the graphics processor, so that the graphics processor performs model training based on the first training data and the second training data.

[0017] In some embodiments, the training data identifier includes a first data identifier and a second data identifier, and the loading module is specifically used to:

[0018] When the first training data corresponding to the first data identifier exists in the local cache, loading the target training data into the graphics processor;

[0019] Sending a second data request to the data cluster, wherein the second data request includes a second data identifier;

[0020] Loading second training data returned by the data cluster based on the second data request into a local memory, where the second training data corresponds to the second data identifier;

[0021] The second training data is loaded from the local memory to the graphics processor.

[0022] In some embodiments, the model training apparatus further comprises:

[0023] The asynchronous loading module is used to asynchronously write the second training data returned by the data cluster based on the second data request into the local cache.

[0024] In some embodiments, the asynchronous loading module is specifically used to:

[0025] When the currently used capacity of the local cache is less than the set cache capacity, the second training data returned by the data cluster based on the second data request is asynchronously written into the local cache.

[0026] In some embodiments, the model training apparatus further comprises:

[0027] A parameter receiving module, used for receiving cache configuration parameter information;

[0028] A parameter saving module, used for saving the cache configuration parameter information after the cache configuration parameter information passes verification;

[0029] The cache establishment module is used to establish a local cache based on the cache configuration parameter information.

[0030] In some embodiments, the parameter storage module includes a detection submodule and a storage submodule, wherein:

[0031] A detection submodule, used to detect whether a model training start message is received after the cache configuration parameter information passes verification;

[0032] The saving submodule is used to save the cache configuration parameter information when a model training start message is received.

[0033] In some embodiments, the preservation submodule is specifically used for:

[0034] When a model training start message is received, a parameter saving request is generated;

[0035] Sending the parameter saving request to the data system to trigger the data system to save the cache configuration parameter information;

[0036] At this time, the cache establishment module is specifically used to:

[0037] The data system is triggered to establish a local cache based on the cache configuration parameter information.

[0038] In some embodiments, the cache configuration parameter information includes a cache directory and a cache capacity, and the model training device further includes:

[0039] The verification module is used to determine that the cache configuration parameter information passes the verification when the cache directory already exists locally and the cache capacity does not exceed the local available threshold.

[0040] In some embodiments, the model training apparatus further comprises:

[0041] The cache deletion module is used to delete the local cache when a model training termination message is received.

[0042] In some embodiments, the cache deletion module is specifically used to:

[0043] When a model training termination message is received, a cache destruction request is generated;

[0044] The cache destruction request is sent to the data system to trigger the data system to delete the local cache. Accordingly, the embodiment of the present application also provides a storage medium, the storage medium stores a computer program, and the computer program is suitable for loading by a processor to execute any model training method provided in the embodiment of the present application.

[0045] Correspondingly, an embodiment of the present application also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, any one of the model training methods provided in the embodiment of the present application is implemented.

[0046] The present application can obtain a data request, which includes a training data identifier required for model training; query the target training data corresponding to the training data identifier from the local cache, and the local cache is used for storage; when the target training data corresponding to the training data identifier exists in the local cache, the target training data is loaded into the graphics processor, so that the graphics processor performs model training based on the target training data; when the target training data corresponding to the training data identifier does not exist in the local cache, the target training data is obtained from the data cluster, and the target training data is loaded into the graphics processor, so that the graphics processor performs model training based on the target training data.

[0047] When receiving a data request, the present application can query the target training data required for model training from the local cache of the computer device. When the target training data exists in the local cache, the target training data in the local cache can be loaded into the image processor for model training. Compared with the prior art that needs to read from a remote storage cluster, the present application can effectively shorten the time spent on reading training data, thereby effectively improving the efficiency of model training. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0049] Figure 1 It is a scenario diagram of a model training system provided in an embodiment of the present application;

[0050] Figure 2 It is a flowchart of the model training method provided in the embodiment of the present application;

[0051] Figure 3 is another flow chart of the model training method provided in the embodiment of the present application;

[0052] Figure 4 It is a partial implementation diagram of the model training method provided in the embodiment of the present application;

[0053] Figure 5 It is another partial implementation schematic diagram of the model training method provided in the embodiment of the present application;

[0054] Figure 6 It is a schematic diagram of the implementation of the model training method provided in the embodiment of the present application;

[0055] Figure 7 It is a structural schematic diagram of a model training device provided in an embodiment of the present application;

[0056] Figure 8 It is a schematic diagram of the structure of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0057] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the embodiments described in this application are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of this application.

[0058] Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that machines have the functions of perception, reasoning and decision-making.

[0059] Machine Learning (ML) is a multi-disciplinary subject that involves probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory and other disciplines. It specializes in studying how computers simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications are spread across all areas of artificial intelligence. Machine learning and deep learning usually include artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and self-learning.

[0060] The model training method in the present application can be widely used in the field of artificial intelligence, such as neural network model training and learning model training in the field of machine learning. For example, in the fields of computer vision, speech technology, natural language processing, etc. of artificial intelligence, the model training method in the present application can be applicable to the training process of models built to solve specific problems, etc.

[0061] The model training method of the present application can be integrated into a model training system, which can be integrated into one or more computer devices, which can include a terminal or a server, etc., wherein the server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides cloud computing services. The terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, etc., but is not limited to this. The terminal and the server can be directly or indirectly connected via wired or wireless communication, and this application does not limit this.

[0062] See also Figure 1 The model training system may include a model training device, wherein the model training device may obtain a data request, the data request including a training data identifier required for model training; query target training data corresponding to the training data identifier from a local cache, the local cache being used for storage; when the target training data corresponding to the training data identifier exists in the local cache, the target training data is loaded into a graphics processor, so that the graphics processor performs model training based on the target training data; when the target training data corresponding to the training data identifier does not exist in the local cache, the target training data is obtained from a data cluster, and the target training data is loaded into the graphics processor, so that the graphics processor performs model training based on the target training data.

[0063] It should be noted that Figure 1The scenario diagram of the model training system shown is only an example. The model training system and scenario described in the embodiment of the present application are intended to more clearly illustrate the technical solution of the embodiment of the present application, and do not constitute a limitation on the technical solution provided in the embodiment of the present application. Ordinary technicians in this field can know that with the evolution of the model training device and the emergence of new business scenarios, the technical solution provided in the embodiment of the present application is also applicable to similar technical problems.

[0064] In this embodiment, the model training method is described in detail. The model training method can be integrated in a terminal or a server, such as Figure 2 As shown, Figure 2 : is a flow chart of a model training method provided in an embodiment of the present application. The model training method may include:

[0065] 101. Obtain a data request, where the data request includes a training data identifier required for model training.

[0066] In the present application, model training may include a model training process in the field of artificial intelligence, and a trained model that can solve specific problems and achieve specific purposes is obtained through model training. For example, the model trained in the present application may include a speech recognition model, an image detection model, a semantic parsing model, an autonomous driving model, an intelligent question-and-answer model, or a machine translation model, etc. Based on the model training method of the present application, a trained model is obtained, and related problems are solved through the trained model.

[0067] The data request may include a message requesting data required for model training. Therefore, the data request may include a training data identifier. The training data identifier may be a data identifier of the data required for model training. The training data identifier may serve as a unique identifier for the corresponding training data.

[0068] The data request can be issued by a training system that controls the model training process, and the training system that controls the model training process can be integrated with the model training method of the present application on the same computer device; the training system can also be separately integrated on other computer devices, which are different from the computer devices integrated with the model training method of the present application, and so on.

[0069] As the computer devices integrated in the training system differ, the method of obtaining data requests will also change. For example, when the training system and the model training method of the present application are integrated on the same computer device, data requests can be obtained by simply transmitting information within the computer device. For another example, when the training system and the model training method of the present application are integrated on different computer devices, it is necessary to use wired or wireless connection methods and corresponding communication technologies to receive data requests sent by other computer devices, and so on.

[0070] For example, the neural network model is trained by a model training device, and the model training device may receive a data request S, which may include a training data identifier T of data required for training the neural network model.

[0071] 102. Query the target training data corresponding to the training data identifier from the local cache, where the local cache is used for storage.

[0072] The present application can perform model training through a model training device, and the model training method of the present application can be integrated on the model training device. The model training device can be a computer device, such as a terminal or a server.

[0073] Among them, the local cache may include a high-speed accessible memory on a computer device that integrates the model training method, such as a high-speed memory in a model training device.

[0074] Among them, the target training data may include data required for model training, the target training data corresponds to the training data identifier, and the target training data may include multiple forms, such as text, image, video, voice, etc.

[0075] The amount of data required for the entire model training process is extremely large, so the training data is usually stored in a remote data cluster, such as a server. The model training process requires multiple trainings, and the data required each time is different. It takes a lot of time to obtain the target training data from the remote data cluster before each training, which makes the entire model training process time-consuming.

[0076] For example, the target training data M corresponding to the training data identifier T can be queried from the local cache, and the local cache is a high-speed memory on the model training device.

[0077] In some embodiments, the model training apparatus further comprises:

[0078] Receive cache configuration parameter information; save the cache configuration parameter information after the cache configuration parameter information passes verification; and establish a local cache based on the cache configuration parameter information.

[0079] The present application can establish a local cache in a computer device so that target training data can be stored and read at high speed in the local cache during model training, thereby effectively improving the efficiency of model training.

[0080] Among them, the cache configuration parameter information may include parameter-related information of the local cache to be established, for example, the cache configuration parameter information may include cache directory information, cache capacity information, etc. The cache configuration parameter information may be manually input by the user, or the cache configuration parameter information may be automatically determined based on the target training data required for model training, such as automatically determining the cache configuration parameter information based on the amount of target training data required for a round of model training, etc.

[0081] After receiving the cache configuration parameter information, the cache configuration parameter information can be verified to check whether the cache configuration parameter information meets the set requirements, where the set requirements may include hard requirements limited by the computer device itself, or may include individual requirements of a specific model training process, etc., which can be flexibly set and are not limited here. When the cache verification parameter information passes the verification, the cache verification parameter information can be saved to the disk of the computer device for subsequent reading and use.

[0082] Before the local cache is needed, the stored model configuration parameter information can be read, and based on the model configuration parameter information, a local cache can be established on the computer device.

[0083] For example, cache configuration parameter information 1 is received, and after cache configuration parameter information 1 passes verification, cache configuration parameter information 1 is saved to the disk of the model training device, and finally, based on cache configuration parameter information 1, a local cache is established on the model training device.

[0084] In some embodiments, the step of “saving the cache configuration parameter information after the cache configuration parameter information passes verification” includes:

[0085] After the cache configuration parameter information passes verification, it is detected whether the model training startup message is received; when the model training startup message is received, the cache configuration parameter information is saved.

[0086] In a computer device, a cache can be a memory that implements high-speed data exchange, but the cache is used in many scenarios. In order to ensure the effective use of the cache in the computer device, the local cache used for model training in the present application can be established at a specific time so that the local cache can be used soon after it is established, avoiding unnecessary occupation of the cache. For example, the present application can establish a local cache at the beginning of model training or before the beginning of model training.

[0087] In order to save disk storage resources, cache configuration parameter information can be saved at the beginning of model training or before model training begins. In addition, when special circumstances such as power failure and restart of computer equipment occur, cache configuration parameter information can be read from the disk to quickly rebuild the local cache.

[0088] Among them, the model training startup message can indicate that model training is about to begin. The model training startup message can have different content and different sources in different model training scenarios. Specifically, the model training startup message can be issued based on the user's operation, or it can be automatically generated and triggered by the system.

[0089] In some scenarios, the model training startup message may include messages with specific functions, such as container creation instructions. Since these messages with specific functions indicate that model training is about to begin, these messages can be designated as model training startup messages, such as container creation instructions, which can indicate that model training is about to begin.

[0090] Therefore, after the cache configuration parameter information passes verification, when a model training start message is received, the cache configuration parameter information can be saved, and a local cache can be established based on the cache configuration parameter information.

[0091] In some embodiments, the step of “saving cache configuration parameter information when receiving a model training start message” includes:

[0092] When a model training start message is received, a parameter saving request is generated; the parameter saving request is sent to the data system, triggering the data system to save cache configuration parameter information;

[0093] At this time, the step of "establishing a local cache based on cache configuration parameter information" may include:

[0094] The trigger data system establishes a local cache based on the cache configuration parameter information.

[0095] In the present application, related operations on the training data can be implemented through the data system, and the data system can be integrated with the model training method of the present application on the same computer device. Specifically, when a model training startup message is received, a parameter save request can be generated based on the model training startup message, and the parameter save request can be sent to the data system. The data system can save the cache configuration parameter information to the disk of the computer device based on the parameter save request.

[0096] The data system can also create and delete local caches. For example, the data system can create a local cache based on saved cache configuration parameter information.

[0097] For example, when the model training start message 1 is received, a parameter save request 1 can be generated and sent to the data system S. The data system S can save the cache configuration parameter information in the disk of the model training device based on the data save request 1, and establish a local cache based on the cache configuration parameter information.

[0098] In some embodiments, the cache configuration parameter information includes a cache directory and a cache capacity, and the model testing method further includes:

[0099] When the cache directory already exists locally and the cache capacity does not exceed the local available threshold, it is determined that the cache configuration parameter information passes the verification.

[0100] In order to ensure that the local cache is successfully established, the cache configuration parameter information can be verified before performing the establishment operation, where the cache configuration parameter information may include a cache directory and cache capacity, the cache directory may include address information of the local cache to be established in the computer device, and the cache capacity may include storage capacity information occupied by the local cache to be established in the computer device cache.

[0101] The local available threshold may be flexibly set according to the actual application scenario. For example, the local available threshold may include the upper limit of the available capacity of the computer device cache, or may include an artificially set upper limit of the local cache capacity, and so on.

[0102] Specifically, the verification may include checking whether the cache directory already exists locally on the computer device, and whether the cache capacity is greater than a local available threshold. When both conditions are met, it can be determined that the cache configuration parameters have passed the verification.

[0103] For example, the cache configuration parameter information includes cache directory 1 and cache capacity 1. When cache directory 1 already exists locally and cache capacity 1 does not exceed the local available threshold Y, it can be determined that the cache configuration parameter information passes the verification.

[0104] 103. When target training data corresponding to the training data identifier exists in the local cache, the target training data is loaded into the graphics processor, so that the graphics processor performs model training based on the target training data.

[0105] The present application can query the target training data in a computer device (such as a model training device) that integrates the model training method of the present application. When the target training data exists, the target training data in the local cache can be directly read out and loaded into the graphics processor of the computer device. The image processor inputs the target training data into the model and calculates to obtain the output result. The parameters in the model are then updated in combination with the data labels and other content carried by the target training data to complete a round of model training.

[0106] The present application can first query the target training data in the local cache of the computer device. When the data exists, only data flow is required within the computer device, and the target training data in the local cache is loaded into the graphics processor to perform model training. Compared with the method of obtaining target training data from the server, the present application can save a lot of time, thereby effectively improving the efficiency of model training.

[0107] For example, when the target training data M exists in the local cache, the target training data can be loaded into the local memory of the model training device, and then loaded into the graphics processor of the model training device. The graphics processor trains the neural network model to be trained using the target training data M.

[0108] 104. When the target training data corresponding to the training data identifier does not exist in the local cache, the target training data is obtained from the data cluster, and the target training data is loaded into the graphics processor, so that the graphics processor performs model training based on the target training data.

[0109] In the present application, when the target training data is not found in the local cache, a training data acquisition request can be sent to the data cluster, the training data acquisition request includes a training data identifier, and then the target training data returned by the data cluster based on the data acquisition request is received, and the target training data is loaded into the graphics processor, so as to perform model training in the image processor through the target training data.

[0110] Specifically, the process of acquiring target training data from a data cluster may also be implemented through a data system, for example, by sending a training data acquisition request including a training data identifier to the data cluster through the data system, and receiving the target training data returned by the data cluster based on the data acquisition request.

[0111] In some embodiments, the training data identifier includes a first data identifier and a second data identifier, and the step of "when target training data corresponding to the training data identifier exists in the local cache, loading the target training data into the graphics processor so that the graphics processor performs model training based on the target training data" may include:

[0112] When first training data corresponding to a first data identifier exists in a local cache, the first training data is loaded into a graphics processor, and second training data corresponding to a second data identifier obtained from a data cluster is loaded into the graphics processor, so that the graphics processor performs model training based on the first training data and the second training data.

[0113] The amount of target training data required to complete a round of model training is huge. Therefore, the data request may include multiple training data identifiers. The training data stored in the local cache may include a part of the target training data corresponding to the training data identifier. This part of the target training data in the local cache can be loaded into the graphics processor, and then the other part of the target training data obtained from the data cluster can be loaded into the graphics processor, so as to perform model training based on the target training data through the graphics processor.

[0114] For example, the training data identifier may include a first data identifier and a second data identifier, the training data identifier corresponds to target training data, the first data identifier corresponds to the first training data, the second data identifier corresponds to the second training data, and the target training data may include the first training data and the second training data. Then, when the first training data or the second training data exists in the local cache, the first training data or the second training data may be loaded into the graphics processor, and the second training data or the first training data that does not exist in the local cache and is obtained from the data cluster may be loaded into the graphics processor, so that the model training is performed based on the first training data and the second training data through the graphics processor.

[0115] In some embodiments, the step of “loading the second training data corresponding to the second data identifier obtained from the data cluster to the graphics processor” may include:

[0116] Send a second data request to the data cluster, the second data request includes a second data identifier; load the second training data returned by the data cluster based on the second data request into the local memory, the second training data corresponding to the second data identifier; load the second training data from the local memory to the graphics processor.

[0117] For example, a second data request 1 is sent to the data server, the second data request 1 includes a second data identifier 1, the data server returns second training data based on the second data request 1, loads the returned second training data into the local memory of the model training device, and then loads the second training data from the local memory of the model training device to the graphics processor.

[0118] In some embodiments, the model testing method further comprises:

[0119] Asynchronously writing the second training data returned by the data cluster based on the second data request into the local cache.

[0120] During multiple rounds of model training, the target training data required for each round is a part of all the training data, and all the training data remains unchanged. For example, the training data identifier required for each round of model training can be determined by sampling the identifiers of all the training data. Based on the fact that the target training data of each round of model training accounts for a fixed proportion of all the training data and that there are many rounds of model training, the target training data or part of the target training data obtained from the data cluster can be written into the local cache starting from the first round of model training. In this way, when obtaining the target training data for the next round, it can be read from the local cache that has stored part of the all the training data, and then the part of the target training data that does not exist in the local cache can be obtained from the data cluster.

[0121] Specifically, the target training data or part of the target training data obtained from the data cluster can be written into the local cache in an asynchronous manner. At the same time, the computer device can also perform model training based on the target training data. The two are processed in parallel, effectively improving the efficiency of the entire model training process.

[0122] For example, the data server returns the second training data based on the second data request 1, and can asynchronously write the second training data into the local cache of the model training machine.

[0123] In some embodiments, the step of “asynchronously writing the second training data returned by the data cluster based on the second data request into the local cache” may include:

[0124] When the currently used capacity of the local cache is less than the set cache capacity, the second training data returned by the data cluster based on the second data request is asynchronously written into the local cache.

[0125] Since the local cache is set with a set cache capacity during configuration, when the current used capacity of the local cache is greater than the set cache capacity, the target training data or part of the target training data (such as the second training data in this embodiment) can no longer be written to the local cache. When the current used capacity of the local cache is less than or equal to the set cache capacity, the target training data or part of the target training data (such as the second training data in this embodiment) can be written to the local cache.

[0126] In some embodiments, the model testing method further comprises:

[0127] When the model training termination message is received, the local cache is deleted.

[0128] In computer devices, cache can realize storage of high-speed data exchange, but cache is used in many scenarios. In order to ensure the effective use of cache in computer devices, the local cache used for model training in this application can be deleted at specific times to avoid unnecessary occupation of cache.

[0129] In the present application, the local cache may be deleted when it is no longer of use value, such as when the model training is terminated. The termination of the model training may be determined by a model training termination message. Therefore, the local cache may be deleted, for example, when the model training termination message is received, the local cache is deleted.

[0130] Among them, the model training termination message can indicate that the model training has been terminated. The model training termination message can have different content and different sources in different model training scenarios. For example, the model training termination message can be sent manually, or it can be automatically generated and sent by the system, etc. For example, the model training termination message can be a message corresponding to the end of training midway or the completion of training. In some scenarios, the model training termination message can include a message indicating that the training is midway or the training is completed, such as a container destruction instruction indicating that the training is completed. When the container destruction instruction is received, the local cache can be deleted.

[0131] In some embodiments, the step of “deleting the local cache upon receiving a model training termination message” may include:

[0132] When a model training termination message is received, a cache destruction request is generated;

[0133] Send a cache destruction request to the data system, triggering the data system to delete the local cache.

[0134] Specifically, the local cache can be deleted through the data system. For example, when a model training termination message is received, a cache destruction request can be generated and sent to the data system, so that the data system deletes the local cache.

[0135] For example, when the model training termination message 1 is received, a cache destruction request 1 can be generated and sent to the data system S. The data system S can delete the established local cache based on the cache destruction request 1.

[0136] When receiving a data request, the present application can query the target training data required for model training from the local cache of the computer device. When the target training data exists in the local cache, the target training data in the local cache can be loaded into the image processor for model training. Compared with the prior art that needs to read from a remote storage cluster, the present application can effectively shorten the time spent on reading training data, thereby effectively improving the efficiency of model training.

[0137] The method described in the above embodiment is further described in detail below with examples.

[0138] This application will introduce the model training method integrated in the model training device as an example. The model training device can be a computer device, such as Figure 3 As shown, Figure 3 : is a flow chart of a model training method provided in an embodiment of the present application. The model training method may include:

[0139] 201. The computer device receives cache configuration parameter information.

[0140] For example, cache configuration parameter information may include cache directory and cache capacity customized by the user based on actual model training requirements.

[0141] 202. When the cache configuration parameter information passes verification and a model training start message is received, the computer device saves the cache configuration parameter information.

[0142] For example, see Figure 4 , you can specify the cache directory and space size (i.e., cache capacity) parameter configuration (as specified by the user), and then check whether the configuration parameters are normal. The check may include determining whether the cache directory already exists and whether the cache capacity exceeds the available free threshold of the local cache of the computer device. If the cache directory already exists and the cache capacity does not exceed the available free threshold, it can be determined that the cache configuration parameter information has passed the check. If the cache directory does not exist, or the cache capacity exceeds the available free threshold, it can be determined that the cache configuration parameters have not passed the check, and a prompt message needs to be returned to instruct the user to reconfigure the cache directory and cache capacity.

[0143] The user can submit a model training task and create a task container (pod) according to the needs of the model training task (i.e., the model training startup message). At this time, the cache configuration verification parameters can be passed to the client of the storage system on the computer device (i.e., the fuse client), and the fuse client saves the cache configuration parameter information in the local disk of the computer device (i.e., the fuse client executes the passed-in parameters). After the cache configuration parameter information is saved to the local disk, when the client of the storage system is restarted, etc., the client of the storage system can directly establish a local cache based on the saved cache configuration parameter information, which effectively improves the stability of the model training process.

[0144] 203. The computer device establishes a local cache based on the cache configuration parameter information.

[0145] For example, the client of the cache system establishes a local read-only cache on the computer device based on the cache directory and cache capacity. The data stored in the local read-only cache cannot be modified, ensuring the consistency of the training data stored in the local read-only cache during the entire model training process, thereby effectively ensuring the smooth progress of the model training process.

[0146] 204. The computer device obtains a first-round data request, where the first-round data request includes a first-round data identifier required for a first-round model training.

[0147] 205. The computer device obtains the first round of training data corresponding to the training data identifier from the data cluster, and loads the first round of training data into the graphics processor, so that the graphics processor performs a first round of model training based on the first round of training data.

[0148] For example, the model training process of a model to be trained may include multiple rounds. Before the first round of model training, the first round of training data can be obtained from the storage system. The storage system may include a data cluster for storing training data remotely. The first round of training data corresponding to the first round of training identifier can be read from the remote data cluster, and the first round of training data can be loaded into the computer device memory, and then loaded from the memory into the graphics processor to perform the first round of model training.

[0149] 206. The computer device asynchronously writes the first round of training data into the local cache.

[0150] For example, when reading the first round of training data from a remote data cluster of the storage system, the first round of training data can be asynchronously written to the local read-only cache of the computer device, so that the subsequent training process can first search from the local read-only cache. The use of asynchronous writing can enable the first round of model training process and the data writing process to the local read-only cache to be processed in parallel, thereby effectively improving the efficiency of the entire model training process.

[0151] 207. The computer device obtains a training data request, where the training data request includes a training data identifier required for model training, and the training data identifier includes a first data identifier and a second data identifier.

[0152] 208. When first training data corresponding to the first data identifier exists in the local cache of the computer device, the computer device loads the first training data into the graphics processor, and loads second training data corresponding to the second data identifier obtained from the data cluster into the graphics processor, so that the graphics processor performs model training based on the first training data and the second training data.

[0153] After the first round of model training is completed, the local read-only cache contains the first round of training data, which is part of all the training data required for the entire model training process. All the training data required for the entire model training process is constant. The training data required for each round of model training is extracted from all the training data. Therefore, part of the training data required for each round of model training is the data that has been used in the previous round.

[0154] Based on this, in the present application, during each round of model training including the first round of model training, if the available capacity of the local read-only cache is not zero, the training data required for the current round read from the remote data cluster can be written into the local read-only cache. Then, in each round of model training, the target training data required for the current round can be queried from the local read-only cache first. When part of the target training data exists in the local read-only cache, another part of the target training data can be obtained from the remote data cluster, thereby effectively saving model training time and improving model training efficiency.

[0155] For example, the process of obtaining target training data can be found in Figure 5 First, read the AI ​​training data from the local disk read-only cache. When the local disk has data, you can load the data into the memory to run AI training. When the local disk does not have data, you can read the required AI training data from the Ceph remote data cluster, write the data asynchronously to the local disk cache, and load the data into the memory to run AI training.

[0156] 209. When receiving the model training termination message, the computer device deletes the local cache.

[0157] For example, when the entire model training process is completed or ends midway, the container (pod) will be destroyed. At the time of container destruction, a local read-only cache cleanup instruction can be transmitted to the client of the storage system. The client can delete the local read-only cache according to this instruction to complete the recovery of the cache space of the computer device.

[0158] The entire model training process can be found in Figure 6 The data reading process can include AI training sending a data reading request to the client of the storage system (such as ceph-fuse), the client of the storage system (such as ceph-fuse) reading the training data from the ceph cluster on the cloud, the remote storage cluster (such as ceph cluster) transmitting the training data for AI training, and the client of the storage system (such as ceph-fuse) asynchronously caching the training data to the local disk (such as SSD solid state drive). The local read-only cache can be created by the client of the storage system based on the cache directory and space size at the time of pod creation, and the local read-only cache can be deleted by the client of the storage system at the time of pod destruction (deleting the training data and the local read-only cache).

[0159] When receiving a data request, the present application can query the target training data required for model training from the local cache of the computer device. When the target training data exists in the local cache, the target training data in the local cache can be loaded into the image processor for model training. Compared with the prior art that needs to read from a remote storage cluster, the present application can effectively shorten the time spent on reading training data, thereby effectively improving the efficiency of model training.

[0160] In order to facilitate better implementation of the model training method provided in the embodiment of the present application, the embodiment of the present application also provides a device based on the above model training method. The meanings of the nouns are the same as those in the above model training method, and the specific implementation details can refer to the description in the method embodiment.

[0161] like Figure 7 As shown, Figure 7A schematic diagram of the structure of a model training device provided in an embodiment of the present application, wherein the model training device may include an acquisition module 301, a query module 302, a loading module 303 and a cluster acquisition module 304, wherein:

[0162] An acquisition module 301 is used to acquire a data request, where the data request includes a training data identifier required for model training;

[0163] A query module 302, used to query the target training data corresponding to the training data identifier from the local cache, and the local cache is used for storage;

[0164] The loading module 303 is used to load the target training data into the graphics processor when the target training data corresponding to the training data identifier exists in the local cache, so that the graphics processor performs model training based on the target training data.

[0165] The cluster acquisition module 304 is used to acquire the target training data from the data cluster when the target training data corresponding to the training data identifier does not exist in the local cache, and load the target training data to the graphics processor, so that the graphics processor performs model training based on the target training data.

[0166] In some embodiments, the training data identifier includes a first data identifier and a second data identifier, and the loading module is specifically used to:

[0167] When first training data corresponding to a first data identifier exists in a local cache, the first training data is loaded into a graphics processor, and second training data corresponding to a second data identifier obtained from a data cluster is loaded into the graphics processor, so that the graphics processor performs model training based on the first training data and the second training data.

[0168] In some embodiments, the training data identifier includes a first data identifier and a second data identifier, and the loading module is specifically used to:

[0169] When the first training data corresponding to the first data identifier exists in the local cache, the target training data is loaded into the graphics processor;

[0170] Sending a second data request to the data cluster, where the second data request includes a second data identifier;

[0171] Loading second training data returned by the data cluster based on the second data request into the local memory, the second training data corresponding to the second data identifier;

[0172] The second training data is loaded from the local memory to the GPU.

[0173] In some embodiments, the model training apparatus further comprises:

[0174] The asynchronous loading module is used to asynchronously write the second training data returned by the data cluster based on the second data request into the local cache.

[0175] In some embodiments, the asynchronous loading module is specifically used to:

[0176] When the currently used capacity of the local cache is less than the set cache capacity, the second training data returned by the data cluster based on the second data request is asynchronously written into the local cache.

[0177] In some embodiments, the model training apparatus further comprises:

[0178] A parameter receiving module, used for receiving cache configuration parameter information;

[0179] A parameter saving module, used to save cache configuration parameter information after the cache configuration parameter information passes verification;

[0180] The cache establishment module is used to establish a local cache based on cache configuration parameter information.

[0181] In some embodiments, the parameter storage module includes a detection submodule and a storage submodule, wherein:

[0182] The detection submodule is used to detect whether the model training start message is received after the cache configuration parameter information passes the verification;

[0183] The saving submodule is used to save cache configuration parameter information when a model training startup message is received.

[0184] In some embodiments, the preservation submodule is specifically used for:

[0185] When a model training start message is received, a parameter saving request is generated;

[0186] Send a parameter saving request to the data system, triggering the data system to save cache configuration parameter information;

[0187] At this time, the cache establishment module is specifically used to:

[0188] The trigger data system establishes a local cache based on the cache configuration parameter information.

[0189] In some embodiments, the cache configuration parameter information includes a cache directory and a cache capacity, and the model training device further includes:

[0190] The verification module is used to determine whether the cache configuration parameter information passes the verification when the cache directory already exists locally and the cache capacity does not exceed the local available threshold.

[0191] In some embodiments, the model training apparatus further comprises:

[0192] The cache deletion module is used to delete the local cache when receiving a model training termination message.

[0193] In some embodiments, the cache deletion module is specifically used to:

[0194] When a model training termination message is received, a cache destruction request is generated;

[0195] Send a cache destruction request to the data system, triggering the data system to delete the local cache.

[0196] In the present application, the acquisition module 301 can obtain a data request, which includes a training data identifier required for model training; the query module 302 can query the target training data corresponding to the training data identifier from the local cache, and the local cache is used for storage; the loading module 303 can load the target training data to the graphics processor when the target training data corresponding to the training data identifier exists in the local cache, so that the graphics processor performs model training based on the target training data, and the cluster acquisition module 304 can obtain the target training data from the data cluster when the target training data corresponding to the training data identifier does not exist in the local cache, and load the target training data to the graphics processor, so that the graphics processor performs model training based on the target training data.

[0197] When receiving a data request, the present application can query the target training data required for model training from the local cache of the computer device. When the target training data exists in the local cache, the target training data in the local cache can be loaded into the image processor for model training. Compared with the prior art that needs to read from a remote storage cluster, the present application can effectively shorten the time spent on reading training data, thereby effectively improving the efficiency of model training.

[0198] In addition, the embodiment of the present application also provides a computer device, which can be a terminal or a server. Figure 8 As shown, it shows a schematic diagram of the structure of the computer device involved in the embodiment of the present application, specifically:

[0199] The computer device may include components such as a processor 401 with one or more processing cores, a memory 402 with one or more computer-readable storage media, a power supply 403, and an input unit 404. Those skilled in the art will appreciate that Figure 8 The computer device structure shown in the figure does not constitute a limitation on the computer device, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently. Among them:

[0200] The processor 401 is the control center of the computer device, and uses various interfaces and lines to connect various parts of the entire computer device. By running or executing software programs and / or modules stored in the memory 402, and calling data stored in the memory 402, the processor 401 performs various functions of the computer device and processes data. Optionally, the processor 401 may include one or more processing cores; preferably, the processor 401 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user pages and application programs, etc., and the modem processor mainly processes wireless communications. It is understandable that the above-mentioned modem processor may not be integrated into the processor 401.

[0201] The memory 402 can be used to store software programs and modules. The processor 401 executes various functional applications and data processing by running the software programs and modules stored in the memory 402. The memory 402 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area may store data created according to the use of the computer device, etc. In addition, the memory 402 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices. Accordingly, the memory 402 may also include a memory controller to provide the processor 401 with access to the memory 402.

[0202] The computer device also includes a power supply 403 for supplying power to various components. Preferably, the power supply 403 can be logically connected to the processor 401 through a power management system, so as to manage charging, discharging, and power consumption through the power management system. The power supply 403 can also include any components such as one or more DC or AC power supplies, recharging systems, power failure detection circuits, power converters or inverters, and power status indicators.

[0203] The computer device may further include an input unit 404, which may be used to receive input digital or character information and generate keyboard, mouse, joystick, optical or trackball signal input related to user settings and function control.

[0204] Although not shown, the computer device may further include a display unit, etc., which will not be described in detail herein. Specifically in this embodiment, the processor 401 in the computer device will load the executable files corresponding to the processes of one or more application programs into the memory 402 according to the following instructions, and the processor 401 will run the application programs stored in the memory 402, thereby realizing various functions, as follows:

[0205] A data request is obtained, where the data request includes a training data identifier required for model training; target training data corresponding to the training data identifier is queried from a local cache, where the local cache is used for storage; when the target training data corresponding to the training data identifier exists in the local cache, the target training data is loaded into a graphics processor, so that the graphics processor performs model training based on the target training data; when the target training data corresponding to the training data identifier does not exist in the local cache, the target training data is obtained from a data cluster, and the target training data is loaded into the graphics processor, so that the graphics processor performs model training based on the target training data.

[0206] The specific implementation of the above operations can be found in the previous embodiments, which will not be described in detail here.

[0207] According to one aspect of the present application, a computer program product or a computer program is provided, the computer program product or the computer program including computer instructions, the computer instructions being stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the methods provided in various optional implementations in the above-mentioned embodiments.

[0208] Those skilled in the art will appreciate that all or part of the steps in the various methods of the above embodiments may be completed by a computer program, or by controlling related hardware through a computer program. The computer program may be stored in a computer-readable storage medium and loaded and executed by a processor.

[0209] To this end, the embodiment of the present application further provides a storage medium in which a computer program is stored, and the computer program can be loaded by a processor to execute the steps in any model training method provided in the embodiment of the present application. For example, the computer program can execute the following steps:

[0210] A data request is obtained, where the data request includes a training data identifier required for model training; target training data corresponding to the training data identifier is queried from a local cache, where the local cache is used for storage; when the target training data corresponding to the training data identifier exists in the local cache, the target training data is loaded into a graphics processor, so that the graphics processor performs model training based on the target training data; when the target training data corresponding to the training data identifier does not exist in the local cache, the target training data is obtained from a data cluster, and the target training data is loaded into the graphics processor, so that the graphics processor performs model training based on the target training data.

[0211] The storage medium may include: a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0212] Since the computer program stored in the storage medium can execute the steps in any model training method provided in the embodiments of the present application, the beneficial effects that can be achieved by any model training method provided in the embodiments of the present application can be achieved. Please refer to the previous embodiments for details and will not be repeated here.

[0213] The above is a detailed introduction to a model training method, device, storage medium and computer equipment provided in the embodiments of the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for technical personnel in this field, according to the ideas of the present application, there will be changes in the specific implementation methods and application scopes. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A model training method, characterized in that: include: Receive cache configuration parameter information; When a model training start message is received, the cache configuration parameter information is saved, and a local cache is established based on the cache configuration parameter information; Obtaining a data request, wherein the data request includes a training data identifier required for model training; Querying the target training data corresponding to the training data identifier from the local cache, wherein the local cache is used for storage; When the target training data corresponding to the training data identifier exists in the local cache, the target training data is loaded into the graphics processor, so that the graphics processor performs model training based on the target training data; When the target training data corresponding to the training data identifier does not exist in the local cache, acquiring the target training data from the data cluster, and loading the target training data into the graphics processor, so that the graphics processor performs model training based on the target training data; When a model training termination message is received, the local cache is deleted.

2. The method according to claim 1, characterized in that The training data identifier includes a first data identifier and a second data identifier, and when target training data corresponding to the training data identifier exists in the local cache, the target training data is loaded into a graphics processor so that the graphics processor performs model training based on the target training data, including: When first training data corresponding to the first data identifier exists in the local cache, the first training data is loaded into the graphics processor, and second training data corresponding to the second data identifier obtained from the data cluster is loaded into the graphics processor, so that the graphics processor performs model training based on the first training data and the second training data.

3. The method according to claim 2, characterized in that The step of loading the second training data corresponding to the second data identifier obtained from the data cluster into the graphics processor includes: Sending a second data request to the data cluster, wherein the second data request includes a second data identifier; Loading second training data returned by the data cluster based on the second data request into a local memory, where the second training data corresponds to the second data identifier; The second training data is loaded from the local memory to the graphics processor.

4. The method according to claim 3, characterized in that The method further comprises: Asynchronously writing the second training data returned by the data cluster based on the second data request into a local cache.

5. The method according to claim 4, characterized in that The step of asynchronously writing the second training data returned by the data cluster based on the second data request into the local cache includes: When the currently used capacity of the local cache is less than the set cache capacity, the second training data returned by the data cluster based on the second data request is asynchronously written into the local cache.

6. The method according to claim 1, characterized in that When the model training start message is received, the cache configuration parameter information is saved, and the method further includes: After the cache configuration parameter information passes verification, it is detected whether a model training start message is received.

7. The method according to claim 6, characterized in that When the model training start message is received, the cache configuration parameter information is saved, including: When a model training start message is received, a parameter saving request is generated; Sending the parameter save to the data system, triggering the data system to save the cache configuration parameter information; The establishing of a local cache based on the cache configuration parameter information includes: The data system is triggered to establish a local cache based on the cache configuration parameter information.

8. The method according to claim 6, characterized in that The cache configuration parameter information includes a cache directory and a cache capacity, and the method further includes: When the cache directory already exists locally and the cache capacity does not exceed the local available threshold, it is determined that the cache configuration parameter information passes the verification.

9. The method according to claim 1, characterized in that: When receiving the model training termination message, deleting the local cache includes: When a model training termination message is received, a cache destruction request is generated; The cache destruction request is sent to the data system, triggering the data system to delete the local cache.

10. A model training device, characterized in that: include: A parameter receiving module, used for receiving cache configuration parameter information; A cache establishment module, when receiving a model training start message, saves the cache configuration parameter information and establishes a local cache based on the cache configuration parameter information; An acquisition module, used to acquire a data request, wherein the data request includes a training data identifier required for model training; A query module, used to query the target training data corresponding to the training data identifier from a local cache, wherein the local cache is used for storage; A loading module, configured to load the target training data into a graphics processor when target training data corresponding to the training data identifier exists in the local cache, so that the graphics processor performs model training based on the target training data; A cluster acquisition module, configured to acquire the target training data from a data cluster when the target training data corresponding to the training data identifier does not exist in the local cache, and load the target training data into a graphics processor, so that the graphics processor performs model training based on the target training data; The cache deletion module is used to delete the local cache when a model training termination message is received.

11. A storage medium, characterized in that: The storage medium stores a plurality of computer programs, and the computer programs are suitable for being loaded by a processor to execute the steps in any one of the methods according to claims 1 to 9.

12. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 9 are implemented.

Images