Task processing method and system for model training
By introducing the collaborative work of verification and allocation components in the model training framework, and using temporary tokens for resource allocation, the problem of poor data security in the existing technology is solved, and fine control of user access rights and security guarantee of task data is realized. , reduces development costs and improves system availability.
Patent Information
- Application Number
- CN202410983925.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-22
- Publication Date
- 2024-11-22
AI Technical Summary
The existing model training framework has rough management in terms of data security and is prone to stolen or attacked by malicious users, resulting in high risk of leaks, and the additional security system development costs and poor availability.
By working together between verification components, allocation components and storage components, verify client and cluster credentials with the verification database, generate temporary tokens for resource allocation and access control, ensuring the security and availability of task data.
It realizes fine control of user access rights in the distributed model training framework, prevents illegal access, ensures logical isolation and absolute confidentiality of task data, reduces development costs, and improves training efficiency and system availability.
Smart Images

Figure CN119004098A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the technical field of model training, and in particular to a task processing method and system for model training. Background Art
[0002] Model training refers to the process of using actual data to train learning models developed by various algorithms so that the output of the learning model gradually approaches the actual data, thereby achieving the purpose of accurately predicting actual usage scenarios.
[0003] In the existing model training process, with the widespread application of big data, distributed model training frameworks (such as XLearning) are becoming more and more common. By integrating data storage, resource scheduling, task execution, status monitoring, etc., the storage system stores large-scale data sets. After the client submits the training task to the computing system, the computing system schedules the computing resources and allocates computing resources for the training task. The computing server downloads data from the storage system to execute the training task.
[0004] However, the existing model training framework only controls the client's login permissions. On the one hand, after submitting a training task, the client can arbitrarily access and modify the task data of other training tasks. The management is rough and it is very easy to be stolen by malicious users or attacked by external users, resulting in frequent leakage, theft and abuse of task data, which seriously endangers the data security of the training system and poses a serious risk of leakage. On the other hand, the development cost of additional security systems is high, interrupting training tasks and affecting actual production, and cannot be integrated with the training framework, resulting in poor usability. Summary of the invention
[0005] In view of this, the embodiments of the present disclosure provide a task processing method and system for model training, which can solve the problems of rough management, easy theft by malicious users, or attack by external users, resulting in frequent leakage, theft and abuse of task data, seriously endangering the data security of the training system, and posing a serious risk of leakage; the development cost of additional security systems is high, interrupting training tasks affects actual production, and cannot be integrated with the training framework and has poor usability.
[0006] To achieve the above objective, according to one aspect of the present disclosure, a task processing method for model training is provided, the task processing method is applied to a verification component, and the task processing method includes:
[0007] Receive an execution request for one or more training tasks initiated by a client; wherein the execution request includes a task identifier, a client identifier, and a cluster identifier of a service cluster;
[0008] Verifying the client identifier and the cluster identifier using a verification database to obtain a verification result;
[0009] Obtain a first password matching the client identifier and a second password matching the cluster identifier from the verification database, encrypt the verification result and the task identifier using the first password and the second password respectively to obtain a first ciphertext and a second ciphertext, and return the first ciphertext and the second ciphertext to the client.
[0010] According to another aspect of the present disclosure, a task processing system for model training is provided, comprising a verification component, a distribution component, an execution component and a storage component, wherein:
[0011] The verification component receives an execution request for one or more training tasks initiated by a client; wherein the execution request includes a client identifier and a cluster identifier of the client;
[0012] The verification component verifies the client identifier and the cluster identifier using a verification database to obtain a verification result;
[0013] The verification component obtains a first password matching the client identifier and a second password matching the cluster identifier from the verification database, encrypts the verification result and the task identifier using the first password and the second password respectively to obtain a first ciphertext and a second ciphertext, and returns the first ciphertext and the second ciphertext to the client;
[0014] The allocation component receives a resource allocation request initiated by the client and carrying an identity credential and the second ciphertext; wherein the identity credential is obtained by the client decrypting the first ciphertext using a client password;
[0015] The allocation component decrypts the second ciphertext using the local cluster password, and verifies whether the verification result in the decryption result is consistent with the identity credential;
[0016] When the verification result in the decryption result is consistent with the identity credential, the allocation component initiates a token generation request to the storage component according to the identity credential, the client identifier and the task identifier in the decryption result;
[0017] The storage component determines whether the identity credential included in the token generation request allows access, and if the identity credential allows access, responds to the token generation request and issues a temporary token to the allocation component;
[0018] In response to the resource allocation request, the allocation component sends the resource allocation instruction carrying the temporary token to the execution component;
[0019] In response to the resource allocation instruction, the execution component determines the training resources required for the training task according to the task type corresponding to the task identifier and the data volume of the task data;
[0020] The execution component determines one or more target execution nodes of the training task according to the training resources, and sends the temporary token to the target execution nodes.
[0021] According to another aspect of the present disclosure, there is provided an electronic device, comprising:
[0022] Processor; and
[0023] Memory for storing programs,
[0024] Among them, the program includes instructions, which, when executed by the processor, enable the processor to perform the task processing method of the model training.
[0025] According to another aspect of an embodiment of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute the task processing method of the model training.
[0026] One or more technical solutions provided in the embodiments of the present application can achieve the management of user access rights in a distributed model training framework by controlling user identity credentials and data access tokens in a distributed training framework, realize fine-grained control of task granularity, prevent illegal access, modification and other operations, ensure logical isolation and absolute confidentiality of task data, greatly reduce development costs, improve training efficiency while ensuring system data security, and achieve technical effects of high availability and stable operation. BRIEF DESCRIPTION OF THE DRAWINGS
[0027] Further details, features and advantages of the present disclosure are disclosed in the following description of exemplary embodiments in conjunction with the accompanying drawings, in which:
[0028] Figure 1 A flowchart of a task processing method for model training according to an exemplary first embodiment of the present disclosure is shown;
[0029] Figure 2 A schematic diagram of a task processing system for model training according to an exemplary embodiment of the present disclosure is shown;
[0030] Figure 3 A flowchart of a verification method of a verification component according to an exemplary embodiment of the present disclosure is shown;
[0031] Figure 4 A flowchart of a registration method of a client or a service cluster according to an exemplary embodiment of the present disclosure is shown;
[0032] Figure 5 A flowchart of a task processing method for model training according to an exemplary second embodiment of the present disclosure is shown;
[0033] Figure 6 A flowchart of a task processing method for model training according to an exemplary third embodiment of the present disclosure is shown;
[0034] Figure 7 A flowchart of a task processing method for model training according to an exemplary fourth embodiment of the present disclosure is shown;
[0035] Figure 8 A flowchart of a task processing method for model training according to an exemplary fifth embodiment of the present disclosure is shown;
[0036] Fig. 9 A structural block diagram of an exemplary electronic device that can be used to implement the embodiments of the present disclosure is shown. DETAILED DESCRIPTION
[0037] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments described herein, which are instead provided for a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.
[0038] It should be understood that the various steps described in the method embodiments of the present disclosure may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this respect.
[0039] The term "including" and its variations used in this document are open inclusions, that is, "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one other embodiment"; the term "some embodiments" means "at least some embodiments". Relevant definitions of other terms will be given in the description below. It should be noted that the concepts of "first", "second", etc. mentioned in this disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.
[0040] It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".
[0041] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.
[0042] XLearning: A lightweight distributed model training framework that supports the training of multiple machine learning models and is easy to use.
[0043] HDFS: Hadoop Distributed File System, a distributed file system that can split large files into multiple small files and store them on multiple servers.
[0044] Aspects of the present disclosure are described below with reference to the accompanying drawings.
[0045] Figure 1 A flowchart of a task processing method for model training according to an exemplary first embodiment of the present disclosure is shown. Figure 1 As shown, the task processing method for model training in the first embodiment of the present disclosure includes the following steps:
[0046] In the embodiments of the present disclosure, the task processing method of the model training of each embodiment of the present disclosure is executed by each component of the task processing system of the model training, such as Figure 2 As shown, the task processing system for model training disclosed in the present invention includes a client, a verification component, an allocation component, an execution component and a storage component. The verification component includes a verification database, which is used to determine whether the client has access rights to the training task and return the verification result; if the client has the initiation authority, the execution component can be subsequently scheduled by the allocation component to execute the training task; the allocation component is used to verify whether the identity credentials sent by the client are valid, and request a temporary token from the storage component for the client that passes the verification, and at the same time connect the temporary token to the execution component, so that the execution component allocates computing resources for the training task requested by the client; after receiving the temporary token, the execution component schedules the execution node to execute the training task through the resource management node, and uses the monitoring node to monitor the execution status of each execution node; the storage component includes a task database, which is used to respond to the temporary token of the execution node and return the corresponding task data.
[0047] Furthermore, the task processing method of the model training of the first embodiment of the present disclosure is executed by the verification component.
[0048] Step S101, receiving an execution request for one or more training tasks initiated by a client; wherein the execution request includes a client identifier and a cluster identifier of a service cluster.
[0049] In the disclosed embodiment, the verification component receives a training task execution request initiated by a client, the execution request includes a client identifier that initiates the execution request, and a cluster identifier of a service cluster that the client requests to execute the training task, for example, Figure 2 As shown, the client identifiers are c1, c2, ..., cn, and the cluster identifiers are sc1, sc2, ..., scn.
[0050] Step S102: Verify the client identifier and the cluster identifier using a verification database to obtain a verification result.
[0051] In the disclosed embodiment, the verification component uses the registration information of each client and service cluster stored in the verification database to determine whether each client has access rights to the service cluster, or whether each client can request the corresponding service cluster to perform a training task, and whether the corresponding service cluster stores the task data required for the training task requested by the client. The registration information includes the client identifier and client password of each client, and the cluster identifier and cluster password of each service cluster.
[0052] Furthermore, if Figure 3 As shown, the verification method of the verification component of the present disclosure includes the following steps:
[0053] Step S301, matching the client identifier and the cluster identifier with the registration information of the verification database, and determining whether the registration information includes the client identifier and the cluster identifier. If yes, go to step S302; if not, go to step S304.
[0054] In the disclosed embodiment, the verification database stores the registration information of the client, the registration information of the service cluster, and the allowed access relationship list between the client and the service cluster, etc. The registration information is the aforementioned registration identifier, registration password, etc. The allowed access relationship list indicates the service clusters that the client can access, such as Figure 4 As shown, the registration method of the client or service cluster disclosed in the present invention includes the following steps:
[0055] Step S401, receiving one or more registration requests; wherein the registration request includes the identification to be registered and the password to be registered of the client and / or the service cluster.
[0056] In the disclosed embodiment, the password to be registered is set by a user of the client and / or the service cluster, or may be randomly generated by the client and / or the service cluster.
[0057] Step S402: Use the client's pending registration password and pending registration identifier as the client's registration identifier and registration password, and the service cluster's pending registration password and pending registration identifier as the service cluster's registration identifier and registration password, and store the corresponding relationship between the registration identifier and the registration password in the verification database.
[0058] In the disclosed embodiment, the to-be-registered identifier of the client that meets the registration requirements is used as the registration identifier of the client, and the to-be-registered password is used as the registration password of the client, and the corresponding relationship between the registration identifier and the registration password of the client is stored in the verification database; the to-be-registered identifier of the service cluster that meets the registration requirements is used as the registration identifier of the service cluster, and the to-be-registered password is used as the registration password of the service cluster, and the corresponding relationship between the registration identifier and the registration password of the service cluster is stored in the verification database. Among them, the registration requirements include whether the characters of the to-be-registered identifier and the to-be-registered password are recognizable, whether the character type belongs to a preset type, or whether the number of digits is less than or equal to the number of digits threshold, etc.
[0059] In the embodiments of the present disclosure, through the registration method of the client or service cluster of the present disclosure, the client and service cluster are registered before receiving the execution request of the training task, so that the ownership of the access rights can be determined based on the registration information later to ensure the confidentiality of the task data, prevent the leakage of task data, and ensure the data security of the model training system.
[0060] Furthermore, the verification component first compares the client identifier, the cluster identifier, and the registration identifier of the verification database to determine whether the registration information includes the client identifier and the cluster identifier of the execution request.
[0061] Step S302, according to the client identifier and the cluster identifier, search the allowed access relationship list of the verification database to determine whether the client corresponding to the client identifier has access rights to the service cluster corresponding to the cluster identifier. If yes, go to step S303; if not, go to step S304.
[0062] In the disclosed embodiment, when the client identifier and the cluster identifier match the registration information successfully, the verification component determines whether there is a corresponding relationship between the client identifier and the cluster identifier in the allowed access relationship list.
[0063] Step S303: determine that the verification result is that access is allowed.
[0064] In the embodiment of the present disclosure, when there is a correspondence between the client identifier and the cluster identifier in the access permission relationship list, it is determined that the verification result is access permission.
[0065] Step S304: determine that the verification result is no access permission.
[0066] In the embodiment of the present disclosure, when the matching result of at least one of the client identifier and the cluster identifier is a match failure, or there is no corresponding relationship between the client identifier and the cluster identifier in the allowed access relationship list, the verification result is determined to be no access right.
[0067] Further, or, in the case that the matching result of at least one of the client identifier and the cluster identifier is a matching failure, the verification component directly refuses to execute the request.
[0068] In the embodiment of the present disclosure, the verification method of the verification component of the present disclosure is used to determine whether the client has access rights to the service cluster, and a verification result is obtained, thereby determining whether the client can perform training tasks through the service cluster to ensure the security of training data, prevent task data from being tampered with or leaked, and avoid the risk of leakage of the model training system.
[0069] Step S103, obtaining a first password matching the client identifier and a second password matching the cluster identifier from the verification database, encrypting the verification result and the task identifier using the first password and the second password respectively to obtain a first ciphertext and a second ciphertext, and returning the first ciphertext and the second ciphertext to the client.
[0070] In the disclosed embodiment, the verification component searches for registration information in the verification database according to the client identifier and the cluster identifier, obtains a first password matching the client identifier and a second password matching the cluster identifier, encrypts the task identifier and the verification result with the first password to obtain a first ciphertext, and encrypts the task identifier and the verification result with the second password to obtain a second ciphertext, which is sent to the client for subsequent training task processing. The encryption algorithm can be selected as needed, and the verification component, the client, and the allocation component can agree in advance.
[0071] In the embodiment of the present disclosure, through the task processing method for model training of the first embodiment of the present disclosure, the client that initiates the execution request of the training task is verified to determine whether it has been registered and has access rights to the corresponding service cluster, and the verification result of the client is obtained. The ciphertext is returned after encryption, so that other components of the task processing system for model training can respond to the execution request to process the training task, ensure the security of the task data, prevent malicious tampering and leakage, and enhance the risk prevention capability of the task processing system.
[0072] In the embodiments of the present disclosure, Figure 5 As shown, the task processing method for model training in the second embodiment of the present disclosure includes the following steps:
[0073] In an embodiment of the present disclosure, the task processing method for model training of the second embodiment of the present disclosure is executed by the client.
[0074] Step S501, initiating an execution request for one or more training tasks to a verification component; wherein the execution request includes a client identifier and a cluster identifier.
[0075] Step S502: Receive the first ciphertext and the second ciphertext returned by the verification component.
[0076] In the disclosed embodiment, the client receives the first ciphertext and the second ciphertext returned by the verification component.
[0077] Step S503: Decrypt the first ciphertext using the local client password to obtain the identity certificate of the client.
[0078] In the disclosed embodiment, after the client successfully registers, the registration password is stored as the client password in the local password book, so that after receiving the first ciphertext, the client password is obtained from the local password book to decrypt the first ciphertext to obtain the client's identity credentials. Accordingly, the client's identity credentials include the task identifier and the verification result.
[0079] Step S504: Initiate a resource allocation request to an allocation component of a service cluster corresponding to the cluster identifier, carrying the identity credential and the second ciphertext.
[0080] In the disclosed embodiment, the client carries the decrypted identity certificate and the second ciphertext, and initiates a resource allocation request to the allocation component of the service cluster, which performs task processing after verification.
[0081] In an embodiment of the present disclosure, through the task processing method of model training of the second embodiment of the present disclosure, after the client receives the ciphertext returned by the verification component, it decrypts it to obtain its own identity credentials, and initiates a resource allocation request to the allocation component with the identity credentials, requesting the allocation component to allocate computing resources for the training task to realize the processing of the training task.
[0082] In the embodiments of the present disclosure, Figure 6 As shown, the task processing method for model training in the third embodiment of the present disclosure includes the following steps:
[0083] In the embodiment of the present disclosure, the task processing method for model training of the third embodiment of the present disclosure is executed by the allocation component of the service cluster.
[0084] Step S601: receiving a resource allocation request initiated by one or more clients carrying identity credentials and a second ciphertext; wherein the resource allocation request includes a client identifier.
[0085] Step S602, decrypt the second ciphertext using the local cluster password, and verify whether the verification result in the decryption result is consistent with the identity credential. If yes, go to step S603; if not, go to step S605.
[0086] In the disclosed embodiment, after the service cluster is successfully registered, the registration password is stored as the cluster password in the local password book, so that after receiving the second ciphertext, the cluster password is obtained from the local password book to decrypt the second ciphertext to obtain the decryption result. Accordingly, the decryption result includes the task identifier and the verification result.
[0087] Furthermore, the allocation component compares the decryption result with the identity credentials carried in the resource allocation request initiated by the client to determine whether the two are consistent. If the two are consistent, it means that the client is a secure client and can access the task data in the task database of the service cluster to perform training tasks, thereby preventing the task data from being maliciously tampered with or leaked, reducing the risk of leakage, and ensuring the system data security of the task processing system for model training.
[0088] Step S603: Initiate a token generation request to a storage component according to the identity credential, the client identifier and the task identifier in the decryption result.
[0089] In the disclosed embodiment, when it is determined that the decryption result is consistent with the identity credential, the allocation component initiates a token generation request based on the identity credential, the client identifier and the task identifier in the decryption result, and applies for a temporary token from the storage component to facilitate access to the task database, thereby improving the efficiency of accessing the task data and the execution efficiency of the model training task while ensuring the security and confidentiality of the task data, and preventing the consumption of processing resources such as verification, encryption and decryption of each access to the data. The task database includes multiple data groups storing task data and data correspondence lists, each of which stores different task data, including loan amount, repayment amount, repayment period, address, credit score, flow, browsing history, order record, collection record, etc.; the data correspondence list stores the correspondence between the task type of various training tasks and the data group identifier of the data group to which the task data required for training belongs. After determining the task type based on the task identifier, the data group identifier of the data group to which the task data required for the training task belongs can be determined, and then a token generation request is initiated based on the identity credential, the client identifier, the task identifier and one or more data group identifiers.
[0090] Step S604: receiving the temporary token returned by the storage component, and in response to the resource allocation request, sending the resource allocation instruction carrying the temporary token to the execution component.
[0091] In the disclosed embodiment, the temporary token includes an accessible data group identifier corresponding to the task identifier. After obtaining the temporary token, the allocation component sends a resource allocation instruction carrying the temporary token to the execution component. The resource allocation instruction includes the task identifier, so that the execution component allocates computing resources to the execution request initiated by the client. The execution node corresponding to the computing resource carries the temporary token to access the task database and collect task data to execute the training task.
[0092] Step S605: reject the resource allocation request.
[0093] In the disclosed embodiment, when it is determined that the decryption result and the identity credential are inconsistent, indicating that the client has no right to access the task data of the service cluster, or that the client and the service cluster have a disagreement due to malicious attacks after verification and before resource allocation, the allocation component directly rejects the resource allocation request to protect the task processing system for model training.
[0094] In the embodiment of the present disclosure, through the task processing method of model training of the third embodiment of the present disclosure, the client's identity credentials and the decryption result of the second ciphertext are verified to determine whether the client is threatened by malicious attacks, so as to ensure the data security of the system and avoid the occurrence of leaks.
[0095] In the embodiments of the present disclosure, Figure 7 As shown, the task processing method for model training in the fourth embodiment of the present disclosure includes the following steps:
[0096] In the embodiment of the present disclosure, the task processing method of the model training of the fourth embodiment of the present disclosure is executed by the storage component of the service cluster. The storage component may be HDFS.
[0097] Step S701, receiving a token generation request; wherein the token generation request includes identity credentials.
[0098] Step S702, determine whether the identity credential allows access, if yes, go to step S703; if no, go to step S704.
[0099] In the disclosed embodiment, the storage component judges the identity credentials to determine whether the client can access the task data of the service cluster.
[0100] Step S703: In response to the token generation request, a temporary token is issued to the allocation component.
[0101] In the disclosed embodiment, when it is confirmed that the identity credentials allow access, a temporary token is issued to the allocation component, so that the allocation component issues a resource allocation instruction to the execution component. After the execution component responds, it uses the temporary token carried in the resource allocation instruction to access the task database.
[0102] Step S704: reject the token generation request.
[0103] In the disclosed embodiment, the storage component rejects the token generation request when confirming that the identity credential is unauthorized to prevent the risk of leakage caused by tampering, theft or attack of task data.
[0104] In the embodiment of the present disclosure, through the task processing method of model training of the fourth embodiment of the present disclosure, the storage component judges the identity credentials requested by the allocation component to determine whether to issue a temporary token to the allocation component to ensure that the execution node corresponding to the client can only access the task data with authority, avoid the risk of leakage of the task system, and ensure the security of task data.
[0105] In the embodiments of the present disclosure, Figure 8 As shown, the task processing method for model training in the fifth embodiment of the present disclosure includes the following steps:
[0106] In an embodiment of the present disclosure, the task processing method for model training of the fifth embodiment of the present disclosure is executed by an execution component of a service cluster, and the execution component includes a resource management node, a monitoring node and multiple execution nodes. The resource management node is used to schedule the execution nodes to allocate computing resources for training tasks, and the monitoring node is used to monitor the operating status of each execution node to retry, restart, interrupt, pause, replace, etc. the blocked execution nodes.
[0107] Step S801, receiving a resource allocation instruction carrying a temporary token; wherein the resource allocation instruction includes a task identifier.
[0108] In the embodiment of the present disclosure, a resource management node of an execution component receives a resource allocation instruction carrying a temporary token.
[0109] Step S802, in response to the resource allocation instruction, determining the training resources required for the training task according to the task type corresponding to the task identifier and the data volume of the task data.
[0110] In an embodiment of the present disclosure, the resource management node of the execution component stores the amount of computing resources required for various types of training tasks, so that the training resources required for the training task requested by the client can be determined based on the task type corresponding to the task identifier and the amount of task data in the data group corresponding to the temporary token in the task database.
[0111] Step S803: determining one or more target execution nodes of the training task according to the training resources.
[0112] In the embodiment of the present disclosure, the resource management node of the execution component determines the target execution node required for the training task of executing the model training according to the amount of computing resources corresponding to the training resources. The computing resources of each execution node may be the same or different.
[0113] Step S804: issuing a temporary token to the target execution node.
[0114] In the embodiment of the present disclosure, the resource management node of the execution component distributes the temporary token issued by the storage component to each target execution node.
[0115] Step S805: The target execution node carries the temporary token to access the task database and collects the task data corresponding to the data group identifier in the temporary token.
[0116] Step S806: Execute the training task using the collected task data.
[0117] In the disclosed embodiment, each target execution node executes a training task using the collected task data.
[0118] Step S807: Send the execution result of the training task to the storage component so that the storage component stores the execution result.
[0119] In the embodiments of the present disclosure, through the task processing method of model training of the fifth embodiment of the present disclosure, the execution component allocates computing resources for the training task, distributes temporary tokens to each target execution node, so that the target execution node carries the temporary token to collect corresponding task data from the task database of the storage component and executes the training task, thereby ensuring the security of the task data while improving the task training efficiency.
[0120] In the embodiments of the present disclosure, the task processing method of model training in each embodiment of the present disclosure can be applied to any model training framework to ensure the data security of the training tasks.
[0121] The exemplary embodiment of the present disclosure also provides an electronic device, comprising: at least one processor; and a memory connected to the at least one processor in communication. The memory stores a computer program that can be executed by the at least one processor, and the computer program is used to cause the electronic device to perform the method according to the embodiment of the present disclosure when executed by the at least one processor.
[0122] The exemplary embodiments of the present disclosure also provide a non-transitory computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor of a computer, is used to cause the computer to perform the method according to the embodiments of the present disclosure.
[0123] The exemplary embodiments of the present disclosure further provide a computer program product, including a computer program, wherein when the computer program is executed by a processor of a computer, the computer is used to enable the computer to perform the method according to the embodiments of the present disclosure.
[0124] refer to Fig. 9 , a block diagram of an electronic device 900 that can be used as a server or client of the present disclosure will now be described, which is an example of a hardware device that can be applied to various aspects of the present disclosure. The electronic device is intended to represent various forms of digital electronic computer devices, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.
[0125] like Fig. 9 As shown, the electronic device 900 includes a computing unit 901, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 902 or a computer program loaded from a storage unit 908 into a random access memory (RAM) 903. In the RAM 903, various programs and data required for the operation of the device 900 can also be stored. The computing unit 901, the ROM 902, and the RAM 903 are connected to each other via a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904.
[0126] A plurality of components in the electronic device 900 are connected to the I / O interface 905, including: an input unit 906, an output unit 907, a storage unit 908, and a communication unit 909. The input unit 906 may be any type of device capable of inputting information to the electronic device 900, and the input unit 906 may receive input digital or character information, and generate key signal inputs related to user settings and / or function control of the electronic device. The output unit 907 may be any type of device capable of presenting information, and may include, but is not limited to, a display, a speaker, a video / audio output terminal, a vibrator, and / or a printer. The storage unit 904 may include, but is not limited to, a disk, an optical disk. The communication unit 909 allows the electronic device 900 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks, and may include, but is not limited to, a modem, a network card, an infrared communication device, a wireless communication transceiver, and / or a chipset, such as a Bluetooth™ device, a WiFi device, a WiMax device, a cellular communication device, and / or the like.
[0127] The computing unit 901 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 901 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 901 performs the various methods and processes described above. For example, in some embodiments, Figure 1 , Figures 3 to 8 The method can be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as the storage unit 908. In some embodiments, part or all of the computer program can be loaded and / or installed on the electronic device 900 via the ROM 902 and / or the communication unit 909. In some embodiments, the computing unit 901 can be configured to execute by any other suitable means (for example, by means of firmware). Figure 1 , Figures 3 to 8 method.
[0128] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.
[0129] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
[0130] As used in this disclosure, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, apparatus, and / or device (e.g., disk, optical disk, memory, programmable logic device (PLD)) for providing machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal for providing machine instructions and / or data to a programmable processor.
[0131] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).
[0132] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.
[0133] A computer system may include clients and servers. Clients and servers are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship to each other.
Claims
1. A task processing method for model training, characterized in that: The task processing method is applied to the verification component, and the task processing method includes: Receive an execution request for one or more training tasks initiated by a client; wherein the execution request includes a task identifier, a client identifier, and a cluster identifier of a service cluster; Using a verification database to verify the client identifier and the cluster identifier to obtain a verification result; Obtain a first password matching the client identifier and a second password matching the cluster identifier from the verification database, encrypt the verification result and the task identifier using the first password and the second password respectively to obtain a first ciphertext and a second ciphertext, and return the first ciphertext and the second ciphertext to the client.
2. The task processing method according to claim 1, characterized in that: The using the verification database to verify the client identifier and the cluster identifier to obtain a verification result includes: Matching the client identifier and the cluster identifier with the registration information of the verification database to determine whether the registration information includes the client identifier and the cluster identifier; In the case where the client identifier and the cluster identifier exist, searching the allowed access relationship list of the verification database to determine whether the client corresponding to the client identifier has access rights to the service cluster corresponding to the cluster identifier; In a case where a corresponding relationship between the client identifier and the cluster identifier exists in the access-allowed relationship list, it is determined that the verification result is access allowed.
3. The task processing method according to claim 2, characterized in that: Also includes: If the matching result of at least one of the client identifier and the cluster identifier is a match failure, or there is no corresponding relationship between the client identifier and the cluster identifier in the allowed access relationship list, the verification result is determined to be unauthorized access or the execution request is rejected.
4. The task processing method according to claim 1, characterized in that: Before receiving the execution request of one or more training tasks initiated by the client, the method further includes: Receiving one or more registration requests; wherein the registration request includes the identification to be registered and the password to be registered of the client and / or the service cluster; The client's password and identifier to be registered are used as the client's registration identifier and password, and the service cluster's password and identifier to be registered are used as the service cluster's registration identifier and password, and the corresponding relationship between the registration identifier and the registration password is stored in the verification database.
5. A task processing method for model training, characterized in that: The task processing method is applied to the client, and the task processing method includes: Initiate an execution request for one or more training tasks to the verification component; wherein the execution request includes a client identifier and a cluster identifier of the client; Receiving a first ciphertext and a second ciphertext returned by the verification component; Decrypting the first ciphertext using the local client password to obtain the identity credential of the client; A resource allocation request is initiated to an allocation component of a service cluster corresponding to the cluster identifier by carrying the identity credential and the second ciphertext.
6. A task processing method for model training, characterized in that: The task processing method is applied to the allocation component, and the task processing method includes: Receiving a resource allocation request initiated by one or more clients carrying identity credentials and a second ciphertext; wherein the resource allocation request includes a client identifier; Decrypting the second ciphertext using the local cluster password, and verifying whether the verification result in the decryption result is consistent with the identity credential; In the case where the verification result in the decryption result is consistent with the identity credential, initiating a token generation request to the storage component according to the identity credential, the client identifier and the task identifier in the decryption result; The temporary token returned by the storage component is received, and in response to the resource allocation request, the resource allocation instruction carrying the temporary token is sent to the execution component.
7. A task processing method for model training, characterized in that: The task processing method is applied to a resource management node of an execution component, and the task processing method includes: Receiving a resource allocation instruction carrying a temporary token; wherein the resource allocation instruction includes a task identifier; In response to the resource allocation instruction, determining the training resources required for the training task according to the task type corresponding to the task identifier and the data volume of the task data; Determining one or more target execution nodes of the training task according to the training resources; The temporary token is issued to the target execution node.
8. The task processing method according to claim 7, characterized in that: The temporary token includes a data group identifier corresponding to the task identifier; the processing method is applied to a target execution node of an execution component, and the method includes: Accessing a task database with the temporary token to collect task data in the data group corresponding to the data group identifier; The training task is executed using the collected task data, and the execution result of the training task is sent to a storage component so that the storage component stores the execution result.
9. A task processing method for model training, characterized in that: The task processing method is applied to a storage component, and the task processing method includes: Receiving a token generation request; wherein the token generation request includes identity credentials; Determine whether the identity credential allows access, and if the identity credential allows access, issue a temporary token to the allocation component in response to the token generation request.
10. A task processing system for model training, characterized in that: It includes client, verification component, allocation component, execution component and storage component, among which: The verification component receives an execution request for one or more training tasks initiated by a client; wherein the execution request includes a client identifier and a cluster identifier of the client; The verification component verifies the client identifier and the cluster identifier using a verification database to obtain a verification result; The verification component obtains a first password matching the client identifier and a second password matching the cluster identifier from the verification database, encrypts the verification result and the task identifier using the first password and the second password respectively to obtain a first ciphertext and a second ciphertext, and returns the first ciphertext and the second ciphertext to the client; The allocation component receives a resource allocation request initiated by the client and carrying an identity credential and the second ciphertext; wherein the identity credential is obtained by the client decrypting the first ciphertext using a client password; The allocation component decrypts the second ciphertext using the local cluster password, and verifies whether the verification result in the decryption result is consistent with the identity credential; When the verification result in the decryption result is consistent with the identity credential, the allocation component initiates a token generation request to the storage component according to the identity credential, the client identifier and the task identifier in the decryption result; The storage component determines whether the identity credential included in the token generation request allows access, and if the identity credential allows access, responds to the token generation request and issues a temporary token to the allocation component; In response to the resource allocation request, the allocation component sends the resource allocation instruction carrying the temporary token to the execution component; In response to the resource allocation instruction, the execution component determines the training resources required for the training task according to the task type corresponding to the task identifier and the data volume of the task data; The execution component determines one or more target execution nodes of the training task according to the training resources, and sends the temporary token to the target execution nodes.
11. An electronic device, comprising: processor; as well as Memory for storing programs, Wherein, the program includes instructions, which, when executed by the processor, enable the processor to perform the task processing method for model training according to any one of claims 1-9.
12. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to enable the computer to execute the task processing method for model training according to any one of claims 1-9.
Citation Information
Cited By
Hybrid storage method and system for unified metadata layer
CN122019474A
Hybrid storage method and system for unified metadata layer
CN122019474B
