Model training system, method and storage medium

Abstract

The present invention provides a model training system, method, and storage medium, relating to the field of machine learning. The model training system includes a cloud data storage platform and a cloud model training platform. The cloud data storage platform is used to store training data and receive training data call requests. Based on the training data call requests, the platform exports the training data corresponding to the data call instructions to the cloud model training platform. The cloud model training platform receives model training creation instructions, obtains a model to be trained, generates and sends training data call requests to the cloud data storage platform, and uses the training data exported from the cloud data storage platform to train the model to be trained, thereby obtaining a training model. The technical solution of the present invention can reduce the risk of training data leakage.

Description

Technical Field

[0001] The present invention relates to the field of machine learning, and in particular to a model training system, method and storage medium. Background Art

[0002] Deep learning is widely used in fields such as artificial intelligence and computer vision. Deep learning requires model training, during which developers design a specific model and iterate through multiple training cycles using a dataset to achieve a deep learning model that meets the desired requirements. The dataset is crucial in determining whether the trained model's stability and accuracy meet the desired requirements. Datasets can be provided by data providers.

[0003] Currently, users can purchase data download permissions from data providers. Once granted, users can download the data and save it locally. When model training is required, the downloaded data is copied to the model training system to complete the training. However, there is a greater risk of data leakage when downloaded locally. Summary of the Invention

[0004] The present application provides a model training system, method and storage medium that can reduce the risk of training data leakage.

[0005] In the first aspect, the present application provides a model training system, including a cloud data storage platform and a cloud model training platform; the cloud data storage platform is used to store training data, and to receive training data call requests, and export the training data corresponding to the data call instructions to the cloud model training platform according to the training data call requests; the cloud model training platform is used to receive model training creation instructions, obtain the model to be trained, and generate and send training data call requests to the cloud data storage platform, and use the training data exported from the cloud data storage platform to train the model to be trained to obtain a training result model.

[0006] According to the first aspect, in the first possibility of the first aspect, the model training system also includes a retrieval data platform and an authentication center; the cloud data storage platform includes a permission gateway; the retrieval data platform is used to establish a data index table based on the training data provided by the data provider, and is used to receive retrieval instructions, perform data retrieval in the data index table according to the retrieval instructions, and generate retrieval results, and is used to receive data selection instructions for the retrieval results from the user terminal, and initiate an authentication permission request to the authentication center according to the data selection instruction, and the authentication permission request includes a data identifier of the training data; the authentication center is used to receive the authentication permission request, create a data identifier according to the authentication permission request Token, and send the data token to the authority gateway and the user terminal; the cloud model training platform is also used to send a training data call request to the authority gateway, and the training data call request includes the data token sent by the authentication center to the user terminal; the authority gateway is used to establish a first corresponding relationship, the first corresponding relationship is a one-to-one correspondence between the data identifier and the data token, and is used to receive the training data call request, according to the data token in the training data call request, search for the target data identifier in the first corresponding relationship, the target data identifier is the data identifier corresponding to the data token in the training data call request, and is used to export the training data corresponding to the target data identifier to the cloud model training platform.

[0007] According to the first aspect, in the second possibility of the first aspect, the model training system also includes a retrieval data platform and an authentication center; the cloud data storage platform includes an authority gateway and at least one data storage server; the retrieval data platform is used to establish a data index table based on the training data provided by the data provider, and receive retrieval instructions, perform data retrieval in the data index table according to the retrieval instructions, and generate retrieval results, and is used to receive data selection instructions from the user terminal for the retrieval results, and initiate an authentication permission request to the authentication center according to the data selection instruction, the authentication permission request including the data identifier of the training data; the authentication center is used to receive the authentication permission request, create a data token for the data identifier according to the authentication permission request, and send the data token to the authority gateway and the user terminal; the cloud model training platform is also Used to send a training data call request to the authority gateway, the training data call request includes the data token sent by the authentication center to the user terminal; the authority gateway is used to establish a second corresponding relationship, the second corresponding relationship is the corresponding relationship between the data token and the data route, the data route includes the uniform resource locator path of the training data, and is used to receive the training data call request, according to the data token in the training data call request, search for the target data route in the second corresponding relationship, the target data route is the data route corresponding to the data token in the training data call request, and is used to access the target data storage server to export the training data indicated by the target data route in the target data storage server to the cloud model training platform, the target data storage server is the data storage server corresponding to the target data route.

[0008] According to the second possibility of the first aspect, in the third possibility of the first aspect, the model training system also includes an access router, and the authority gateway exports the training data indicated by the target data routing from the target data storage server through a predetermined standard access interface in the access router.

[0009] According to the first possibility or the second possibility of the first aspect, in the fourth possibility of the first aspect, the authority gateway is also used to obtain update judgment parameters, determine whether the update judgment parameters meet the update conditions, and send an update request to the authentication center if it is determined that the update judgment parameters meet the update conditions, and to synchronize the data token with the authentication center; the authentication center is also used to receive the update request and update the data token according to the update request.

[0010] According to the fourth possibility of the first aspect, in the fifth possibility of the first aspect, the update judgment parameter includes the number of rejections of the authentication permission request; the authority gateway is also used to monitor the processing process of the authentication center for the authentication permission request, and to send an update request to the authentication center if it is monitored that the number of rejections of the authentication permission request by the authentication center exceeds the rejection number update threshold in the update condition.

[0011] According to the fifth possibility of the first aspect, in the sixth possibility of the first aspect, the update judgment parameter includes the number of calls of the training data; the authority gateway is also used to obtain the number of calls of the training data within a period of time, and to send an update request to the authentication center if the number of calls of the same training data exceeds the call number update threshold in the update condition within a period of time.

[0012] According to the first aspect, in the seventh possibility of the first aspect, the cloud model training platform is also used to destroy the training data and the model to be trained used in training the training result model in the cloud model training platform after training the training result model.

[0013] According to the first aspect, in the eighth possibility of the first aspect, the model training system also includes a data audit system; the data audit system is used to authenticate the validity of the training data uploaded by the data provider, and refuse to store the training data that fails the validity authentication into the cloud data storage platform.

[0014] According to the first aspect, in the ninth possibility of the first aspect, the model training system also includes a cloud model storage platform; the cloud model storage platform is used to provide the model to be trained and save the training result model.

[0015] According to the ninth possibility of the first aspect, in the tenth possibility of the first aspect, the model training system also includes a mirror platform and a model inference platform; the mirror platform is used to store the model inference operating environment; the model inference platform is used to receive inference requests, the inference requests include the data to be processed, and load the model inference operating environment from the mirror platform, and call the training result model from the cloud model storage platform, and import the data to be processed into the training result model for model inference.

[0016] In the second aspect, the present application provides a model training method, including: a cloud model training platform receives a model training creation instruction and obtains a model to be trained; the cloud model training platform generates and sends a training data call request to a cloud data storage platform to call the training data stored in the cloud data storage platform; the cloud data storage platform receives the training data call request and exports the training data corresponding to the training data call request to the cloud model training platform; the cloud model training platform uses the training data exported from the cloud data storage platform to train the model to be trained and obtain a training result model.

[0017] According to the second aspect, in the first possibility of the second aspect, the above-mentioned model training method also includes: the retrieval data platform establishes a data index table based on the training data provided by the data provider; the retrieval data platform receives the retrieval instruction, performs data retrieval in the data index table according to the retrieval instruction, and generates a retrieval result; the retrieval data platform receives the data selection instruction of the user terminal, and initiates an authentication permission request to the authentication center according to the data selection instruction, and the authentication permission request includes the data identifier of the training data; the authentication center receives the authentication permission request, creates a data token of the data identifier according to the authentication permission request, and sends the data token to the authority gateway and the user terminal; the authority gateway establishes a first corresponding relationship based on the data token sent, and the first corresponding relationship is a one-to-one correspondence between the data identifier and the data token.

[0018] According to the first possibility of the second aspect, in the second possibility of the second aspect, the cloud model training platform generates and sends a training data call request to the cloud data storage platform, including: the cloud model training platform generates and sends a training data call request to the authority gateway, the training data call request includes the data token sent by the authentication center to the user terminal; the cloud data storage platform receives the training data call request, and exports the training data corresponding to the training data call request to the cloud model training platform, including: the authority gateway in the cloud data storage platform receives the training data call request, searches for the target data identifier in the first corresponding relationship according to the data token in the training data call request, and exports the training data corresponding to the target data identifier to the cloud model training platform, the target data identifier is the data identifier corresponding to the data token in the training data call request.

[0019] According to the second aspect, in the third possibility of the second aspect, the above-mentioned model training method also includes: the retrieval data platform establishes a data index table based on the training data provided by the data provider; the retrieval data platform receives the retrieval instruction, performs data retrieval in the data index table according to the retrieval instruction, and generates and sends the retrieval results; the retrieval data platform receives the data selection instruction of the user terminal for the retrieval result, and initiates an authentication permission request to the authentication center according to the data selection instruction, and the authentication permission request includes the data identifier of the training data; the authentication center receives the authentication permission request, creates a data token of the data identifier according to the authentication permission request, and issues the data token to the authority gateway and the user terminal; the authority gateway establishes a second corresponding relationship based on the data token issued, and the second corresponding relationship is the corresponding relationship between the data token and the data route, and the data route includes the uniform resource locator path of the training data.

[0020] According to the third possibility of the second aspect, in the fourth possibility of the second aspect, the cloud model training platform generates and sends a training data call request to the cloud data storage platform, including: the cloud model training platform generates and sends a training data call request to the authority gateway, the training data call request includes the data token sent by the authentication center to the user terminal; the cloud data storage platform receives the training data call request, and exports the training data corresponding to the training data call request to the cloud model training platform, including: the authority gateway in the cloud data storage platform receives the training data call request, and according to the data token in the training data call request, searches for the target data route in the second corresponding relationship, the target data route is the data route corresponding to the data token in the training data call request; the authority gateway accesses the target data storage server to export the training data indicated by the target data route in the target data storage server to the cloud model training platform, and the target data storage server is the data storage server corresponding to the target data route.

[0021] According to the second aspect or any one of the first possibility to the fourth possibility of the second aspect, in the fifth possibility of the second aspect, the above-mentioned model training method also includes: the authority gateway obtains the update judgment parameter, and determines whether the update judgment parameter meets the update condition; if it is determined that the update judgment parameter meets the update condition, the authority gateway sends an update request to the authentication center; the authentication center receives the update request, and updates the data token according to the update request; the authority gateway and the authentication center synchronously update the data token.

[0022] According to the fifth possibility of the second aspect, in the sixth possibility of the second aspect, the update judgment parameter includes the number of rejections of the authentication permission request; the authority gateway obtains the update judgment parameter and determines whether the update judgment parameter meets the update condition, including: the authority gateway monitors the processing process of the authentication center for the authentication permission request, and obtains the number of rejections of the authentication permission request by the authentication center, and determines whether the number of rejections of the authentication permission request by the authentication center exceeds the rejection number update threshold in the update condition; if it is determined that the update judgment parameter meets the update condition, the authority gateway sends an update request to the authentication center, including: if it is monitored that the number of rejections of the authentication permission request by the authentication center exceeds the rejection number update threshold in the update condition, then an update request is sent to the authentication center.

[0023] According to the fifth possibility of the second aspect, in the seventh possibility of the second aspect, the update judgment parameter includes the number of calls of the training data; the authority gateway obtains the update judgment parameter and determines whether the update judgment parameter meets the update condition, including: the authority gateway obtains the number of calls of the training data within a period of time, and determines whether the number of calls of the same training data within the period of time exceeds the number of calls update threshold in the update condition; if it is determined that the update judgment parameter meets the update condition, the authority gateway sends an update request to the authentication center, including: if the number of calls of the same training data within a period of time exceeds the number of calls update threshold in the update condition, then send an update request to the authentication center.

[0024] According to the second aspect, in the eighth possibility of the second aspect, after the cloud model training platform uses the training data exported from the cloud data storage platform to train the model to be trained and obtains the training result model, it also includes: the cloud model training platform destroys the training data and the model to be trained used for training the training result model in the cloud model training platform.

[0025] According to the second aspect, in the ninth possibility of the second aspect, the above-mentioned model training method also includes: the data audit system performs validity authentication on the training data uploaded by the data provider; the data audit system refuses to store the training data that fails the validity authentication into the cloud data storage platform.

[0026] According to the second aspect, in the tenth possibility of the second aspect, after the cloud model training platform uses the training data exported from the cloud data storage platform to train the model to be trained and obtains the training result model, it also includes: the cloud model storage platform saves the training result model.

[0027] According to the tenth possibility of the second aspect, in the eleventh possibility of the second aspect, the above-mentioned model training method also includes: the model inference platform receives an inference request, and the inference request includes the data to be processed; the model inference platform loads the model inference operating environment from the mirror platform, and calls the training result model from the cloud model storage platform, and imports the data to be processed into the training result model for model inference.

[0028] On the third aspect, the present application provides a storage medium on which a program is stored, and when the program is executed by a processor, the model training method in the above technical solution is implemented.

[0029] The present application provides a model training system, method, and storage medium that can be applied in deep learning scenarios. The model training system may include a cloud data storage platform and a cloud model training platform. The cloud data storage platform stores training data. The cloud model training platform receives a user's model training creation instruction and triggers the execution of model training. The cloud model training platform calls the training data stored on the cloud data storage platform by sending a training data call request to the cloud data storage platform. The cloud model training platform uses the acquired model to be trained and the training data exported from the cloud data storage platform to perform model training. In the present application, the cloud data storage platform and the cloud model training platform are independent of each other, separating the two functions of training data storage and model training. The cloud data storage platform and the cloud model training platform are both implemented based on the cloud system. The model training process is carried out in the cloud system. Users performing model training cannot download the training data locally. The training data exists on the cloud data storage platform and the cloud model training platform that is performing model training. In other words, the training data will not be leaked from the local user side, thereby reducing the risk of training data leakage. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 This is a schematic diagram of an application scenario of a model training system according to an embodiment of the present invention;

[0031] Figure 2 This is a schematic diagram of the structure of a model training system in one embodiment of the present invention;

[0032] Figure 3 A schematic structural diagram of a model training system according to another embodiment of the present invention;

[0033] Figure 4 Schematic diagram of the structure of a model training system in another embodiment of the present invention;

[0034] Figure 5 This is a flow chart of a model training method according to one embodiment of the present invention;

[0035] Figure 6 This is a flowchart of a specific implementation of a model training method in one embodiment of the present invention;

[0036] Figure 7 The present invention is a flowchart of another specific implementation of a model training method in one embodiment of the present invention. DETAILED DESCRIPTION

[0037] Embodiments of the present invention provide a model training system, method, and storage medium that can be applied in deep learning scenarios. These systems can train deep learning models and apply them, such as performing inference using trained deep learning models. The model training system of embodiments of the present invention can perform model training and model inference in the cloud. Figure 1 Schematic diagram of the application scenario of the model training system of the embodiment of the present invention. Figure 1 As shown, the model training system can run on a cloud service system, which can be composed of a cloud system and a system cluster gateway that provides an external access interface. Users can connect to the cloud system via a user terminal using their account and password. The cloud system includes multiple servers interconnected by an internal network. The model training system can store and provide training data and training models through a data model warehouse. The model training system can achieve human-computer interaction between the model training system and users through a deep learning database, authenticate various rights between users and the model training system through an authentication service system, and complete model training and inference through a training and inference system.

[0038] Figure 2 FIG. 1 is a schematic diagram of the structure of a model training system according to an embodiment of the present invention. Figure 2 As shown, the model training system includes a cloud data storage platform 11 and a cloud model training platform 12.

[0039] The cloud data storage platform 11 is used to store training data and receive training data call requests. According to the training data call requests, the training data corresponding to the data call instructions are exported to the cloud model training platform 12.

[0040] Training data is the data required to train a model. The cloud data storage platform 11 can store multiple training data sets, which can be considered a dataset consisting of multiple pieces of data. Training data may include images, videos, audio, and other items, but this is not a limitation. When storing training data, the cloud data storage platform 11 can assign a data identifier to the training data. The data identifier is used to identify the training data and can serve as an identifier for locating the data storage location. In one example, to distinguish different training data sets, the data identifier of the training data is unique; that is, different training data sets have different data identifiers.

[0041] The cloud data storage platform 11 can receive training data uploaded by data providers. For example, the data provider can use a client to connect to the cloud system's backend via the Hypertext Transfer Protocol (HTTP), thereby exchanging information with the cloud data storage platform 11. In one example, the cloud data storage platform 11 can provide the data provider with a standard protocol for uploading training data, which may include data format, compression format, and data type. The cloud data storage platform 11 can test the training data uploaded by the data provider. If it determines that the training data uploaded by the data provider does not comply with the standard protocol, the cloud data storage platform 11 can refuse to store the non-compliant training data.

[0042] A backup area may be provided in the cloud data storage platform 11, and the backup area may be used to back up the training data to avoid data accidents, such as data misoperation that may lead to irrecoverable situations.

[0043] The training data call request is generated and sent by the cloud model training platform 12. The training data call request provides information about the training data requested by the cloud model training platform 12. In one example, the training data call request may include a data identifier. Upon receiving the training data call request, the cloud data storage platform 11 may search for the training data requested and export the requested training data to the cloud model training platform 12, allowing the cloud model training platform 12 to use the exported training data for model training.

[0044] The cloud model training platform 12 is used to receive model training creation instructions, obtain the model to be trained, generate and send training data call requests to the cloud data storage platform 11, and use the training data exported from the cloud data storage platform 11 to train the model to be trained and obtain the training result model.

[0045] Among them, the cloud model training platform 12 can obtain the model to be trained uploaded by the user or model provider, and can also obtain the model to be trained from the model database in the cloud system.

[0046] In one example, a user can connect to the backend of the cloud system via the Hypertext Transfer Protocol using a user terminal 20, thereby interacting with the cloud model training platform 12. The user can send a model training creation instruction to the cloud model training platform 12 via the user terminal 20, triggering the cloud model training platform 12 to create a model training task. The cloud model training platform 12 can perform model training using the model to be trained and the training data. For example, model training can involve importing training data into the model to be trained and performing multiple iterative training cycles to obtain a trained model, i.e., a training outcome model.

[0047] It should be noted that the user of the user terminal 20 may include a user, a data provider or a model provider.

[0048] The cloud data storage platform 11 in the embodiment of the present invention can be regarded as Figure 1 The cloud model training platform 12 in the embodiment of the present invention can be regarded as a part of the data model warehouse. Figure 1 Part of training the inference system.

[0049] In this embodiment of the present invention, the cloud data storage platform 11 and the cloud model training platform 12 are independent of each other, separating the functions of training data storage from model training. Both the cloud data storage platform 11 and the cloud model training platform 12 are implemented based on a cloud system. The model training process is performed in the cloud system. Users performing model training cannot download training data locally. The training data is stored on both the cloud data storage platform 11 and the cloud model training platform 12 where the model training is being performed. This means that training data cannot be leaked from local users, thereby reducing the risk of training data leakage.

[0050] Figure 3 Schematic diagram of the structure of a model training system in another embodiment of the present invention. Figure 3 and Figure 2 The difference is that Figure 2 The cloud data storage platform 11 also includes Figure 3 The authority gateway 111 in; Figure 3 The model training system shown may also include a data retrieval platform 13, an authentication center 14, a data audit system 15, a cloud model storage platform 16, a mirror platform 17 and a model reasoning platform 18.

[0051] The data retrieval platform 13 is used to create a data index table based on the training data provided by the data provider. Users can search and query the training data stored in the cloud data storage platform 11 through the data retrieval platform 13.

[0052] In one example, after the data provider uploads the training data, the data retrieval platform 13 can analyze and process the training data to obtain basic data information such as the data set size, data set scale, data owner information, and data upload date of the training data, so that users can understand the basic information of the training data.

[0053] In one example, the cloud data storage platform 11 may also require data providers to provide training data tags when uploading training data. The training data tags may represent the characteristics of the training data. Specifically, the training data tags may be keywords representing the content represented by the training data. For example, when uploading training data, the data provider may tag the training data with the tags "license plate" and "small car." When establishing the data index table, the data retrieval platform 13 may also add the training data tags to the data retrieval table, allowing users to utilize the characteristics of the training data when searching for the training data.

[0054] The data retrieval platform 13 is used to receive retrieval instructions, perform data retrieval in the data index table according to the retrieval instructions, and generate retrieval results. Specifically, the retrieval instruction may include one or more retrieval keywords, and the tags of the training data in the data index table may be searched based on the retrieval keywords. The retrieval results may include information about the training data related to the retrieval keywords in the retrieval instruction, such as the name, number, keywords of the training data, and partial data examples in the training data. In one example, the retrieval results may include information about the training data arranged in order of relevance to the retrieval keywords, allowing users to more intuitively obtain the training data most relevant to the retrieval keywords. In another example, information about fixed-data training data may be randomly selected from the information about the training data retrieved based on the retrieval keywords and provided to the user. For example, the retrieval results generated for each retrieval may include information about ten pieces of training data. The data retrieval platform 13 may send the retrieval results to the user terminal 20, which may display the retrieval results.

[0055] After receiving the search results, the user can also issue a data selection instruction for the search results through the user terminal 20. The data retrieval platform receives the data selection instruction from the user terminal 20 for the search results and initiates an authentication permission request to the authentication center 14 based on the data selection instruction. The data selection instruction can be used to instruct the selection of one or more training data items from the search results, thereby determining the training data required for model training.

[0056] After determining the training data required for model training, an authentication permission request is initiated to the authentication center 14. The authentication permission request may include the data identifier of the training data and request the authentication center 14 for the right to call the training data.

[0057] The retrieval data platform 13 in the embodiment of the present invention can be regarded as Figure 1 At least a portion of the deep learning database in .

[0058] The authentication center 14 is used to receive an authentication permission request, create a data token of a data identifier according to the authentication permission request, and send the data token to the authority gateway 111 and the user terminal 20 .

[0059] The authentication permission request is used to request the right to call the training data. The authentication center 14 can decide whether to agree to the authentication permission request sent by the retrieval data platform 13. Exemplarily, the authentication permission request may include payment information for the training data. If the payment information indicates that the user has successfully paid for the training data, the authentication center 14 may agree to the authentication permission request and create a data token for the data identifier. After the authentication center 14 agrees to the authentication permission request, it may also generate and save data authentication information. The data authentication information may include a user identifier and a data identifier. Exemplarily, the data authentication information may have a valid period, that is, within the valid period, if the user requests the same training data again, the authentication permission request may be directly approved by the authentication center 14 without the need for review. The valid period can be set according to the work scenario and work requirements, and is not limited here. For example, the valid period may be one year or permanent.

[0060] A data token (i.e., a data token) can identify training data in a specific operation and serve as a security credential for data calls. For example, a data token identifies training data in subsequent data call operations. In one example, the data token can be implemented as a security plug-in. The authentication center 14 issues the created data token to the user terminal 20, allowing the user terminal 20 to use the data token to export the training data corresponding to the data token from the cloud data storage platform 11 via the authority gateway 111. The authentication center 14 also stores the created data token.

[0061] The cloud model training platform 12 is also used to send a training data call request to the authority gateway 111 . The training data call request includes a data token sent by the authentication center 14 to the user terminal 20 .

[0062] For example, when requesting training data, the user terminal 20 may add a data token to a model training creation instruction. The cloud model training platform 12 may parse the model training creation instruction, obtain a data token sent to the user terminal 20, and add the data token sent to the user terminal 20 to the training data call request. The cloud model training platform 12 uses the data token in the training data call request to retrieve the training data corresponding to the data token from the cloud data storage platform 11.

[0063] In one implementation, the cloud data storage platform 11 can be implemented as a third-party public server. This third-party public server is independent of the data provider, model provider, or user and is a public server used to store and export training data. Access to training data can be authorized using the correspondence between data tokens and data identifiers.

[0064] The authority gateway 111 is used to establish a first correspondence, which is the correspondence between a data identifier and a data token. The data identifier and the data token correspond one-to-one, and the data token is also unique, that is, different data identifiers correspond to different data tokens. When the authority gateway 111 receives a training data call request, it searches the first correspondence for the target data identifier based on the data token in the training data call request. The target data identifier is the data identifier corresponding to the data token in the training data call request, and the training data corresponding to the target data identifier is exported to the cloud model training platform 12.

[0065] When the cloud data storage platform 11 receives a training data call request, the authority gateway 111 will compare the data token in the training data call request with the data token stored in the authority gateway 111; if the data token in the training data call request can match the data token stored in the authority gateway 111, the training data is allowed to be called, and the training data corresponding to the data token in the training data call request will be exported.

[0066] To ensure data security during model training and prevent unauthorized use of training data, the data token can be updated based on actual circumstances. The authority gateway 111 can be used to obtain update judgment parameters and determine whether the update judgment parameters meet the update conditions. If the update judgment parameters are determined to meet the update conditions, the authority gateway 111 sends an update request to the authentication center 14 and is used to synchronize the data token with the authentication center 14. The authentication center 14 receives the update request and updates the data token based on the update request.

[0067] The update judgment parameters may include one or more of the following parameters: the number of rejections of the authentication permission request, the number of calls of the training data, the existence time of the data token, etc.

[0068] For example, the update judgment parameter includes the number of rejections of the authentication permission request. The authority gateway 111 can monitor the authentication center 14's processing of the authentication permission request, thereby obtaining the number of rejections of the authentication permission request by the authentication center 14. If the authority gateway 111 detects that the number of rejections of the authentication permission request by the authentication center 14 exceeds the rejection number update threshold in the update condition, the authority gateway 111 sends an update request to the authentication center 14.

[0069] The threshold for the number of rejections can be set according to the work scenario and work requirements and is not limited here. The authentication center 14 deletes the original data token, generates a new data token, and sends the new data token to the client terminal and the authority gateway 111 so that the authority gateway 111 can update the data token synchronously with the authentication center 14. When the data token is updated in the authentication center 14 and the authority gateway 111, it is necessary to stop the execution of the training data call request. After the data token in the authentication center 14 and the authority gateway 111 is updated, the training data call request can be executed again. After the data token is updated, if the training data call request still contains the original data token, the original data token in the training data call request will be invalid, and the training data cannot be called.

[0070] For example, the update determination parameter includes the number of training data calls. The authority gateway 111 can obtain the number of training data calls within a certain period of time. If the authority gateway 111 determines that the number of calls for the same training data exceeds the call count update threshold in the update condition within the certain period of time, it will send an update request to the authentication center 14. The period of time for statistical training data and the call count update threshold can be set according to the work scenario and work requirements, and are not limited here.

[0071] For example, the update determination parameter includes the lifetime of the data token. The authority gateway 111 can set the data token's renewal cycle duration and record the data token's lifetime. If the authority gateway 111 determines that the data token's lifetime has reached the renewal cycle duration, it sends an update request to the authentication center 14. The data token's renewal cycle duration can be set based on the work scenario and work requirements and is not limited here.

[0072] It should be noted that the update judgment parameters and update conditions are not limited to the above examples. The authority gateway 111 can also receive the user's update policy configuration instructions and set the update judgment parameters and update conditions according to the update policy configuration instructions.

[0073] The cloud model storage platform 16 is used to provide models to be trained and to store training models. The models stored in the cloud model storage platform 16 can be models uploaded by model providers or training models obtained by the cloud model training platform 12.

[0074] In one example, after the cloud model training platform 12 obtains the training result model through training, it can send the training result model to the cloud model storage platform 16 for storage, and destroy the training data and the model to be trained used in the cloud model training platform 12 for training the training result model. The training result model in the cloud model training platform 12 can also be destroyed to prevent the training data and models left in the cloud model training platform 12, namely the model to be trained and the training result model, from being leaked.

[0075] In one example, the data audit system 15 receives the training data uploaded by the data provider before the cloud data storage platform 11. The data audit system 15 is used to verify the validity of the training data uploaded by the data provider and refuse to store training data that fails the validity verification in the cloud data storage platform 11. For example, if the training data uploaded by the data provider duplicates the training data stored by the cloud data storage platform 11, or the data format of the data uploaded by the data provider does not conform to the standard protocol of the cloud data storage platform 11, the data audit system 15 will determine that the training data uploaded by the data provider is invalid, that is, the uploaded training data fails the validity verification. If the data audit system 15 determines that the training data uploaded by the data provider is valid, it can send a storage instruction to the cloud data storage platform 11 through the retrieval data platform 13, so that the cloud data storage platform 11 will permanently store the training data uploaded by the data provider.

[0076] It should be noted that the method for validating the training data uploaded by the data provider is not limited to the above method. The data audit system 15 can ensure the authenticity and validity of the training data used in the model training system.

[0077] The image platform 17 is used to store the model reasoning operating environment. Specifically, the model reasoning operating environment may include the system environment and the operating framework environment corresponding to the training result model.

[0078] The model inference platform 18 can receive inference requests, which include data to be processed. The inference request can be sent by a user terminal 20. For example, the user terminal 20 can send the inference request to the model inference platform 18 via an application programming interface (API). After receiving the inference request, the model inference platform 18 loads the model inference runtime environment from the image platform 17, calls the trained model from the cloud model storage platform 16, and imports the data to be processed into the trained model for model inference.

[0079] The retrieval data platform in the embodiment of the present invention can be regarded as Figure 1 At least a portion of the deep learning database in the embodiment of the present invention. The authentication center 14 in the embodiment of the present invention can be regarded as Figure 1 At least a part of the authentication service system. The model reasoning platform 18 in the embodiment of the present invention can be regarded as Figure 1 Part of the training inference system.

[0080] Figure 4 This is a structural diagram of a model training system in another embodiment of the present invention. Figure 4 The model training system shown is similar to Figure 3The difference of the model training system shown is that the cloud data storage platform 11 can be implemented as at least one private server of the data provider.

[0081] Under the condition that the cloud data storage platform 11 includes an authority gateway 111 and at least one data storage server 112, i.e., a private server, calling training data can be authorized by utilizing the correspondence between data tokens and data routes.

[0082] The data route may include the Uniform Resource Locator (URL) path of the training data, as well as the data access method and the standard for exporting the training data from the cloud data storage platform 11. When uploading the training data, the data provider may also upload the data route corresponding to the training data to the retrieval data platform 13.

[0083] The data retrieval platform 13 may also perform a validity check on the data route. If the data route is determined to be illegal, the data route will be refused to be stored. For example, if the data retrieval platform 13 determines that the data route is inaccessible or the format of the data route does not meet the preset standards in the model training system, the data route will be refused to be stored. Exemplarily, the data retrieval platform 13 may send a rejection instruction to the authority gateway 111 and the authentication center 14, so that both the authority gateway 111 and the authentication center 14 refuse to store the route data.

[0084] The authority gateway 111 can establish a second correspondence between data tokens and data routes. Exemplarily, the second correspondence can be implemented as a data routing table. Training data has corresponding data routes, with a one-to-one correspondence between training data and data tokens, and a one-to-one correspondence between data tokens and data routes. When the data retrieval platform 13 establishes a data index table, the corresponding data routes can be stored in the authority gateway 111.

[0085] After receiving the training data call request, the authority gateway 111 searches the second correspondence for the target data route based on the data token in the training data call request. The target data route is the data route corresponding to the data token in the training data call request. The authority gateway 111 can access the target data storage server 112 based on the data route corresponding to the data token to export the training data indicated by the target data route in the target data storage server 112 to the cloud model training platform 12. The target data storage server 112 is the data storage server 112 corresponding to the target data route.

[0086] To ensure the security of training data stored in data storage server 112 (i.e., a private server), secure encrypted remote access can be established. In one embodiment, the model training system may also include an access router. The authority gateway 111 exports training data indicated by the target data routing from the target data storage server 112 via a predefined standard access interface in the access router. For example, the standard access interface may be a RESTful access interface, and the path of the RESTful access interface may be used as the data routing.

[0087] In one example, to further ensure the security of training data in data storage server 112, authority gateway 111 may randomly select a data token and verify the legitimacy of the data token. If authority gateway 111 determines that the data token is invalid, it may update the data routing table, i.e., update the second correspondence. Specifically, this may be implemented by updating the data token in the second correspondence.

[0088] Figure 5 FIG. 1 is a flow chart of a model training method in one embodiment of the present invention. The model training method can be applied to the model training system in the above embodiment. Figure 5 As shown, the model training method may include step S201 and step S204.

[0089] In step S201, the cloud model training platform receives a model training creation instruction and obtains a model to be trained;

[0090] In step S202, the cloud model training platform generates and sends a training data call request to the cloud data storage platform to call the training data stored in the cloud data storage platform;

[0091] In step S203, the cloud data storage platform receives the training data call request and exports the training data corresponding to the training data call request to the cloud model training platform;

[0092] In step S204, the cloud model training platform uses the training data exported from the cloud data storage platform to train the model to be trained to obtain a training result model.

[0093] For the description of the above steps S201 to S204, please refer to the relevant description of the cloud model training platform and the cloud data storage platform in the above embodiments.

[0094] In this embodiment of the present invention, the cloud data storage platform and cloud model training platform are independent of each other, separating the functions of training data storage and model training. Both the cloud data storage platform and the cloud model training platform are implemented based on a cloud system. The model training process is carried out in the cloud system. Users performing model training cannot download training data locally. The training data exists on both the cloud data storage platform and the cloud model training platform where the model training is being performed. This means that training data cannot be leaked from local users, thereby reducing the risk of training data leakage.

[0095] Figure 6 FIG. 1 is a flow chart of a specific implementation of a model training method in one embodiment of the present invention. Figure 6 As shown, the model training method may include steps S301 to S315.

[0096] In step 301, the data audit system verifies the validity of the training data uploaded by the data provider.

[0097] In step 302, the data audit system refuses to store the training data that fails the validity verification into the cloud data storage platform.

[0098] In step 303, the data retrieval platform creates a data index table based on the training data provided by the data provider.

[0099] In step 304, the data retrieval platform receives the retrieval instruction, performs data retrieval in the data index table according to the retrieval instruction, and generates a retrieval result.

[0100] In step 305, the data retrieval platform receives the data selection instruction from the user terminal and initiates an authentication permission request to the authentication center according to the data selection instruction.

[0101] The authentication permission request includes the data identifier of the training data.

[0102] In step 306, the authentication center receives the authentication permission request, creates a data token for the data identifier according to the authentication permission request, and sends the data token to the authority gateway and user terminal in the cloud data storage platform.

[0103] In step 307, the authority gateway in the cloud data storage platform establishes a first corresponding relationship based on the data token received.

[0104] The first corresponding relationship is the corresponding relationship between the data identifier and the data token.

[0105] In step 308, the cloud model training platform receives the model training creation instruction and obtains the model to be trained.

[0106] In step 309, the cloud model training platform generates and sends a training data call request to the authority gateway in the cloud data storage platform to call the training data stored in the cloud data storage platform.

[0107] The training data call request includes a data token sent by the authentication center to the user terminal.

[0108] In step 310, the authority gateway in the cloud data storage platform receives the training data call request, searches for the target data identifier in the first correspondence according to the data token in the training data call request, and exports the training data corresponding to the target data identifier to the cloud model training platform.

[0109] The target data identifier is the data identifier corresponding to the data token in the training data call request.

[0110] In step 311, the cloud model training platform uses the training data exported from the cloud data storage platform to train the model to be trained to obtain a training result model.

[0111] In step 312, the cloud model storage platform saves the training result model.

[0112] In step 313, the cloud model training platform destroys the training data and the model to be trained used for training the training result model in the cloud model training platform.

[0113] In step 314 , the model inference platform receives an inference request, where the inference request includes data to be processed.

[0114] In step 315, the model inference platform loads the model inference operating environment from the mirror platform, calls the training result model from the cloud model storage platform, and imports the data to be processed into the training result model for model inference.

[0115] Figure 7 The present invention is a flowchart of another specific implementation of a model training method in one embodiment of the present invention. Figure 7 and Figure 6 The difference is that Figure 6 Step S307 in can be replaced by Figure 7 Step S316 in Figure 6 Step S310 in can be replaced by Figure 7 Steps S317 and S318 in .

[0116] In step S316, the authority gateway in the cloud data storage platform establishes a second corresponding relationship based on the data token received.

[0117] The second corresponding relationship is a corresponding relationship between a data token and a data route, wherein the data route includes a uniform resource locator path of the training data.

[0118] In step S317, the authority gateway in the cloud data storage platform receives the training data call request and searches for the target data route in the second corresponding relationship according to the data token in the training data call request.

[0119] The target data route is the data route corresponding to the data token in the training data call request.

[0120] In step S318, the authority gateway in the cloud data storage platform accesses the target data storage server to export the training data indicated by the target data routing in the target data storage server to the cloud model training platform.

[0121] The target data storage server is a data storage server corresponding to the target data route.

[0122] In one example, data tokens can be updated based on specific scenarios to ensure the security of training data. The authority gateway obtains update judgment parameters and determines whether the update judgment parameters meet the update conditions. If the update judgment parameters meet the update conditions, the authority gateway sends an update request to the authentication center. The authentication center receives the update request and updates the data token accordingly. The authority gateway and the authentication center synchronize the data token updates.

[0123] Exemplarily, the update judgment parameter includes the number of rejections of the authentication permission request. The data token update process may be specifically as follows: the authority gateway monitors the authentication center's processing of the authentication permission request, obtains the number of rejections of the authentication permission request by the authentication center, and determines whether the number of rejections of the authentication permission request by the authentication center exceeds the rejection number update threshold in the update condition; if it is monitored that the number of rejections of the authentication permission request by the authentication center exceeds the rejection number update threshold in the update condition, then an update request is sent to the authentication center.

[0124] Exemplarily, the update judgment parameter includes the number of training data calls. The data token update process can specifically include: the authority gateway obtains the number of training data calls within a certain period of time, determines whether the number of calls to the same training data within the period of time exceeds the call count update threshold specified in the update condition; and if the number of calls to the same training data within the period of time exceeds the call count update threshold specified in the update condition, sends an update request to the authentication center.

[0125] For the description of each step in the above method embodiment, reference may be made to the relevant description in the above system embodiment.

[0126] An embodiment of the present invention may also provide a storage medium having a program stored thereon, which implements the model training method in the above embodiment when the program is executed by a processor.

Claims

1. A model training system providing cloud services, characterized in that: Including cloud data storage platform, cloud model training platform, cloud model storage platform and model inference platform; The cloud data storage platform is used to store training data uploaded by a data provider, wherein the training data is provided with a label, and the label of the training data is used to characterize the content of the training data; The cloud model storage platform is used to store the model to be trained, and the model to be trained is uploaded to the cloud model storage platform by the model provider or user; The cloud model training platform is configured to receive a model training creation instruction input by the user, obtain the model to be trained from the cloud model storage platform according to the model training creation instruction, call the training data stored in the cloud data storage platform when the user successfully pays for the training data, train the model to be trained according to the training data, obtain a training result model, and send the training result model to the cloud model storage platform after obtaining the training result model; The model inference platform is used to call the training result model from the cloud model storage platform and import the data to be processed into the training result model for model inference.

2. The model training system according to claim 1, characterized in that The model training system also includes an authentication center, The authentication center is used to receive an authentication permission request, where the authentication permission request includes payment information for the training data, and is used to determine the authority of the training data.

3. The model training system according to claim 1, characterized in that The system also includes a retrieval data platform, The data retrieval platform is used to obtain information about the training data, where the information about the training data includes any one or all of the following: information about the data owner of the training data and a data upload date of the training data.

4. The model training system according to claim 3, characterized in that The data retrieval platform is also used to: Establishing a data index table according to at least one of the label of the training data and the information of the training data; Receive a search instruction including a search keyword input by the user, and search in the data index table according to the search instruction to generate a search result, wherein the search result includes at least one of information of training data related to the search keyword or a label of the training data related to the search keyword.

5. The model training system according to claim 4, characterized in that The data retrieval platform is further configured to send the retrieval results to a user terminal for display to the user; The data retrieval platform is further configured to receive a data selection instruction for the retrieval result sent by the user terminal, wherein the data selection instruction is configured to instruct the data retrieval platform to determine the training data from the retrieval result.

6. The model training system according to any one of claims 1 to 5, characterized in that: The training data is provided with a data route, and the cloud model training platform calls the training data according to the data route.

7. The model training system according to claim 6, characterized in that The data route includes a uniform resource locator path of the training data.

8. The model training system according to claim 1, characterized in that The model training system also includes a mirroring platform, which is used to store a model reasoning runtime environment, and the model reasoning runtime environment includes a runtime framework environment corresponding to the training achievement model; The model inference platform is also used to load the model inference runtime environment from the mirror platform, and import the data to be processed into the training result model to perform model inference in the model inference runtime environment.

9. The model training system according to claim 2, characterized in that The authentication center is further configured to create a data token when the payment information indicates that the user has successfully paid for the training data; The cloud model training platform is used to call the training data corresponding to the data token and stored in the cloud data storage platform.

10. The model training system according to claim 1, characterized in that: After the user successfully pays for the training data, the training data is approved to be called by the user within a validity period.

11. A model training method, characterized in that: A model training system for providing cloud services, the model training system comprising a cloud data storage platform, a cloud model training platform, a cloud model storage platform, and a model inference platform, the method comprising: The cloud data storage platform stores training data uploaded by a data provider, wherein the training data is provided with a label, and the label of the training data is used to characterize the content of the training data; The cloud model storage platform stores the model to be trained, and the model to be trained is uploaded to the cloud model storage platform by a model provider or a user; The cloud model training platform receives the model training creation instruction input by the user, obtains the model to be trained from the cloud model storage platform according to the model training creation instruction, calls the training data stored in the cloud data storage platform when the user successfully pays for the training data, trains the model to be trained according to the training data, obtains a training result model, and sends the training result model to the cloud model storage platform after obtaining the training result model; The model inference platform calls the training result model from the cloud model storage platform and imports the data to be processed into the training result model for model inference.

12. The model training method according to claim 11, characterized in that: The model training system further includes an authentication center, and the method further includes: The authentication center receives an authentication permission request, where the authentication permission request includes payment information for the training data and is used to determine the authority of the training data.

13. The model training method according to claim 11, characterized in that: The model training system further includes a retrieval data platform, and the method further includes: The data retrieval platform obtains information about the training data, where the information about the training data includes any one or all of the following: information about the data owner of the training data and a data upload date of the training data.

14. The model training method according to claim 13, characterized in that: The method further comprises: The data retrieval platform establishes a data index table according to at least one of the label of the training data and the information of the training data; The data retrieval platform receives a retrieval instruction input by the user including a retrieval keyword, and searches the data index table according to the retrieval instruction to generate a retrieval result, wherein the retrieval result includes at least one of information about training data related to the retrieval keyword or a label of the training data related to the retrieval keyword.

15. The model training method according to claim 14, characterized in that: The method further comprises: The data retrieval platform sends the retrieval results to the user terminal for display to the user; The data retrieval platform receives a data selection instruction for the retrieval result sent by the user terminal, where the data selection instruction is used to instruct the data retrieval platform to determine the training data from the retrieval result.

16. The model training method according to any one of claims 11 to 15, characterized in that: The training data is provided with a data route, and the cloud model training platform calls the training data according to the data route.

17. The model training method according to claim 16, characterized in that: The data route includes a uniform resource locator path of the training data.

18. The model training method according to claim 11, characterized in that: The model training system further includes a mirroring platform for storing a model inference runtime environment, wherein the model inference runtime environment includes a runtime framework environment corresponding to the training outcome model. The method further includes: The model inference platform loads the model inference runtime environment from the mirror platform, imports the data to be processed into the training result model, and performs model inference in the model inference runtime environment.

19. The model training method according to claim 12, characterized in that: The method further comprises: When the payment information indicates that the user has successfully paid for the training data, the authentication center creates a data token; The model inference platform calls the training data corresponding to the data token and stored in the cloud data storage platform.

20. The model training method according to claim 11, characterized in that: After the user successfully pays for the training data, the training data is approved to be called by the user within a validity period.

21. A storage medium, characterized in that The storage medium stores a program, which, when executed by the processor, implements the model training method according to any one of claims 11 to 20.

Images