Model training method and device, equipment and storage medium

[0051] Example 1

[0052] The embodiment of the present invention provides a method for training a model, such as figure 1 As shown, the method specifically includes the following steps:

[0053]Step S101: When a model training instruction is received, attribute information is parsed from the model training instruction, and an algorithm to be trained corresponding to the attribute information is acquired from a preset model library.

[0054] Among them, the attribute information is set in advance according to the business requirements, and the attribute information includes: the type of algorithm, the industry to which the model is used, the business scenario of the industry, the algorithm parameters used to generate the algorithm to be trained, the total number of training times, and the parameters used to represent the The preset weight value of the importance of the training task and the interface path of the object used by the model.

[0055] Specifically, step S101 includes:

[0056] Step A1: parse out the algorithm type from the attribute information;

[0057] Among them, in the man-machine dialogue scenario, the algorithm types include: general classification algorithm, agent customer service algorithm and quality inspection classification algorithm;

[0058] In the practical application of the human-machine dialogue scene, the general classification algorithm is used to determine which type of the preset intentions the target question in the human-machine dialogue belongs to; the agent customer service algorithm is used to classify the intentions belonging to the target type. The corresponding answers are determined for the questions of the QA system; the quality inspection classification algorithm is used to divide the multiple questions according to the preset multiple types of intentions.

[0059] Step A2: Find the algorithm template corresponding to the algorithm type from the preset model library;

[0060] In this embodiment, a general classification algorithm template, an agent customer service algorithm template, and a quality inspection classification algorithm template are set in the model library in advance, so that the corresponding algorithm template can be found from the model library according to the algorithm type.

[0061] Step A3: Parse out algorithm parameters from the attribute information, and add the algorithm parameters to the algorithm template to generate the algorithm to be trained.

[0062] In this embodiment, the algorithm template corresponding to the algorithm type in the attribute information is found from the preset model library by using the attribute information, and the algorithm parameters of the attribute information are added to the algorithm template to generate the algorithm to be trained, which can improve the acquisition of the need for training. the speed of the algorithm to be trained.

[0063] Step S102: Acquire resource information corresponding to the attribute information from a preset training configuration table, and acquire a training file for training the algorithm to be trained through a preset upload interface.

[0064] Specifically, step S102 includes:

[0065] Step B1: Obtain a preset training configuration table; wherein, a one-to-one correspondence between various triples information and multiple resource information is set through the training configuration table, and the triplet information includes the following parameter items: Algorithm type, the industry of the model use object, and the business scenario of the industry;

[0066] In the training configuration table, each row of content represents the correspondence between a triplet of information and a resource information, and the training configuration table contains four columns of content. The first column of content represents the algorithm type, and the second column of content represents the model used The industry of the object, the content in the third column represents the business scenario of the industry to which it belongs, and the content in the fourth column represents the resource information.

[0067] In the actual application of human-machine dialogue scenarios, the industry of the model usage object in the triplet information and the business scenario of the industry are used to indicate which business scenario under which industry the trained model is used, for example: Apply the trained model in the background customer service scenarios in the e-commerce industry or in the credit card service consultation scenarios in the banking industry.

[0068] Step B2: parse out the algorithm type, the industry to which the model uses the object, and the parameter value of the business scenario of the industry from the attribute information, and form the parsed parameter value into reference triplet information;

[0069] Step B3: Determine whether the reference triplet information exists in the training configuration table, if so, obtain resource information corresponding to the reference triplet information from the training configuration table, if not, use Default public resource information.

[0070] In the actual application of the man-machine dialogue scene, the resource information corresponding to the classification algorithm in the training configuration table is set to use resource No. 1, the algorithm type is A1, the industry of the model use object is B1, and the application scenario of the industry is C1. Pool; the resource information corresponding to the classification algorithm whose algorithm type is A1, the industry to which the model user belongs is B1, and the application scenario of the industry is C2 is set to use resource pool No. 2; the algorithm type is A1, and the industry to which the model user belongs is The resource information corresponding to the classification algorithm of B2 is set to use resource pool No. 3; the resource information corresponding to the classification algorithm of algorithm type A2 and the industry to which the model user belongs is B3 is set to use resource pool No. 4, and so on. All classification algorithms are pre-configured in the training configuration table according to the triple information and the corresponding resource information. When the triple information is parsed from the attribute information, the resources corresponding to the triple information are found from the training configuration table. information, which is used as resource information for training the algorithm to be trained.

[0071] In this embodiment, the triplet information is parsed from the attribute information, and the triplet information is used to find the corresponding resource information in the preconfigured training configuration table, as the resource information used for training the algorithm to be trained, Compared with the need to manually configure training resources for the algorithm to be trained in the prior art, it has the effect of automatically configuring resources.

[0072] Step S103: Use the resource information and train the algorithm to be trained through the training file to obtain a training result file.

[0073] Specifically, step S103 includes:

[0074] Step C1: The attribute information, the algorithm to be trained, the resource information and the training file constitute a training task, and the training task is stored in a preset waiting queue;

[0075] In the actual application of the human-machine dialogue scene, a training file consisting of questions in the human-machine dialogue and the intent corresponding to the question is obtained through the preset upload interface, and each line in the training file is used to store a question and a question corresponding to a question. Intent, and there are two columns in the training file, the first column is used to represent the questions in the human-machine dialogue, and the first column is used to represent the intent corresponding to each question in the first column.

[0076] Step C2: when receiving the training instruction, obtain the waiting duration of each training task located in the waiting queue;

[0077] Step C3: parse out a preset weight value used to characterize the importance of the training task from the attribute information of each training task;

[0078] In the actual application of the human-machine dialogue scene, the weight value of each training task is set according to the training requirements, so that the training task with higher weight value is trained in advance, and the weight value corresponding to the training task without pre-training is zero. .

[0079] Step C4: adding the waiting time of each training task to the preset weight value to obtain the priority value of the corresponding training task;

[0080] Step C5: Based on the priority value of each training task, and in descending order, sort each training task in the waiting queue;

[0081] Step C6: Execute each training task in the order of the sorting results to obtain training result files of each training task.

[0082] The generated training result file is a file in pb (protocol buffer, protocol cache) format, which cannot be directly opened to generate the target model. The training result file needs to be added to the model loading component to run the training result file to generate the target model.

[0083] Step S104: Add the training result file to a preset model loading component to instantiate a target model.

[0084] Specifically, step S104 includes:

[0085] Periodically scan the storage area for storing the training result file, and when the training result file is scanned, add the training result file to the model loading component and run the training through the model loading component result file to instantiate the target model;

[0086] Wherein, the model loading component is: Tensorflow Serving component.

[0087] In the actual application of the human-machine dialogue scene, since the training file generated by training the algorithm to be trained cannot be directly opened to generate the target model, the model loading component is used to periodically scan the storage area for storing the training result file. When creating the result file, the training result file is added to the model loading component, and the model loading component runs the code in the training result file according to its own loading characteristics to instantiate the target model.

[0088] In this embodiment, the Tensorflow Serving model loading component is used to load the training result file that cannot be directly opened to obtain the target model, so as to facilitate subsequent deployment of the target model.

[0089] Step S105: Modify the configuration file in the model loading component to deploy the target model into a model usage object.

[0090] Specifically, step S105 includes:

[0091] The interface path of the model usage object is parsed from the attribute information, and the interface path is added to the preset deployment interface of the configuration file in the model loading component, so as to deploy the target model into the model usage object .

[0092] In this embodiment, a deployment interface for deploying the target model is preset in the configuration file of the model loading component, the interface path corresponding to the object using the target model is obtained by parsing the attribute information, and the interface path is added to the configuration file for deployment The corresponding area of ​​the interface to deploy the target model into the model consuming object.

[0093] Further, after step S104, the method further includes:

[0094] Step D1: Receive a verification instruction sent by the verification terminal, and parse out a verification file for verifying the target model from the verification instruction; wherein, the verification file includes: standard input and standard output;

[0095] In the actual application of the human-machine dialogue scene, in order to determine the deployment effect of the trained target model, the target model is verified, and the verification file is obtained by receiving the verification instruction. When no verification file is specified, the original training file of the target model is used as the verification file.

[0096] Step D2: Input all standard inputs in the verification file into the target model in turn to obtain the actual output;

[0097] Step D3: Calculate the verification result according to all actual outputs and all standard outputs in the verification file; wherein, the verification results include: accuracy rate, error rate, and rejection rate;

[0098] Step D4: Send the verification result to the verification terminal.

[0099] In the actual application of the man-machine dialogue scene, the verification result is displayed in the verification terminal in the form of a report, and the report of the verification result shows the accuracy rate, error rate, rejection rate and the target model for each standard input in detail. The actual output, so that the user can adjust the target model according to the verification results.

[0100]In this embodiment, the deployment effect of the target model is determined by verifying the target model again, so that the target model can adjust the adaptability according to the verification result.

[0101] The method, device, device and storage medium for training a model provided by the present invention can find out the algorithm to be trained that needs to be trained through the attribute information in the model training instruction, and find out the corresponding training algorithm through the preset training configuration table and attribute information. The resource information used by the training algorithm to achieve the effect of automatically configuring the training resources; the training files used for training are obtained through the upload interface, which saves the process of manually arranging the training data in the prior art, making the acquisition of training files more efficient; The training algorithm is used to obtain the training tasks of the target model, and the execution order of the training tasks is controlled in the waiting queue according to the priority of the training tasks for training to generate the training result file, which achieves the effect of automatically controlling the execution order of the training tasks; The training result file is added to the model loading component, the target model is instantiated, and the configuration file of the model loading component is modified to deploy the target model to the model using object, which achieves the effect of automatically deploying the target model.

[0102] Embodiment 2

[0103] The embodiment of the present invention provides an apparatus for training a model, such as figure 2 As shown, the device specifically includes the following components:

[0104] The receiving module 201 is configured to, when receiving a model training instruction, parse out attribute information from the model training instruction, and obtain an algorithm to be trained corresponding to the attribute information from a preset model library;

[0105] An uploading module 202, configured to obtain resource information corresponding to the attribute information from a preset training configuration table, and obtain a training file for training the algorithm to be trained through a preset upload interface;

[0106] A training module 203, configured to use the resource information and train the algorithm to be trained through the training file to obtain a training result file;

[0107] The instance module 204 is used for adding the training result file to the preset model loading component to instantiate the target model;

[0108] The deployment module 205 is configured to modify the configuration file in the model loading component to deploy the target model into the model usage object.

[0109] Specifically, the receiving module 201 is used for:

[0110] Parse the algorithm type from the attribute information;

[0111] Find the algorithm template corresponding to the algorithm type from the preset model library;

[0112] Algorithm parameters are parsed from the attribute information, and the algorithm parameters are added to the algorithm template to generate the to-be-trained algorithm.

[0113] Specifically, the uploading module 202 is used for:

[0114] Acquire a preset training configuration table; wherein, a one-to-one correspondence between various triples information and multiple resource information is set through the training configuration table, and the triplet information includes the following parameter items: algorithm type, The industry to which the model uses the object, and the business scenario of the industry to which it belongs;

[0115] From the attribute information, the algorithm type, the industry to which the model is used, and the parameter values ​​of the business scenarios of the industry are parsed, and the parsed parameter values ​​are formed into reference triplet information;

[0116] Judging whether the reference triplet information exists in the training configuration table, if so, obtain resource information corresponding to the reference triplet information from the training configuration table, if not, use the preset public resource information.

[0117] Specifically, the training module 203 is used for:

[0118] The attribute information, the algorithm to be trained, the resource information and the training file constitute a training task, and the training task is stored in a preset waiting queue;

[0119] When receiving the training instruction, obtain the waiting duration of each training task located in the waiting queue;

[0120] A preset weight value used to characterize the importance of the training task is parsed from the attribute information of each training task;

[0121] The priority value of the corresponding training task is obtained by adding the waiting time of each training task to the preset weight value;

[0122] Sort each training task in the waiting queue based on the priority value of each training task and in descending order;

[0123] Execute each training task in the order of the sorted results to obtain the training result file of each training task.

[0124] Specifically, the generating module 204 is used for:

[0125] Periodically scan the storage area for storing the training result file, and when the training result file is scanned, add the training result file to the model loading component and run the training through the model loading component result file to instantiate the target model;

[0126] Wherein, the model loading component is: Tensorflow Serving component.

[0127] Specifically, the deployment module 205 is used for:

[0128] The interface path of the model usage object is parsed from the attribute information, and the interface path is added to the preset deployment interface of the configuration file in the model loading component, so as to deploy the target model into the model usage object .

[0129] Further, the device also includes:

[0130] The verification module is used to receive the verification instruction sent by the verification terminal after the training result file is added to the preset model loading component to instantiate the target model, and parse out the verification instruction from the verification instruction. A verification file for verifying the target model; wherein, the verification file includes: standard input and standard output; all standard inputs in the verification file are sequentially input into the target model to obtain the actual output; according to all The actual output of the verification file and all standard outputs in the verification file are used to calculate the verification result; wherein, the verification result includes: accuracy rate, error rate, and rejection rate; the verification result is sent to the verification terminal.

[0131] Embodiment 3

[0132] This embodiment also provides a computer device, such as a smart phone, a tablet computer, a notebook computer, a desktop computer, a rack server, a blade server, a tower server, or a cabinet server (including independent servers, or A server cluster composed of multiple servers), etc. like image 3 As shown, the computer device 30 in this embodiment at least includes but is not limited to: a memory 301 and a processor 302 that can be communicatively connected to each other through a system bus. It should be pointed out that, image 3 Only computer device 30 is shown having components 301-302, but it should be understood that implementation of all shown components is not required and that more or fewer components may be implemented instead.

[0133] In this embodiment, the memory 301 (that is, a readable storage medium) includes a flash memory, a hard disk, a multimedia card, a card-type memory (eg, SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), Read Only Memory (ROM), Electrically Erasable Programmable Read Only Memory (EEPROM), Programmable Read Only Memory (PROM), Magnetic Memory, Magnetic Disk, Optical Disk, etc. In some embodiments, the memory 301 may be an internal storage unit of the computer device 30 , such as a hard disk or a memory of the computer device 30 . In other embodiments, the memory 301 may also be an external storage device of the computer device 30, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, flash memory card (Flash Card), etc. Of course, the memory 301 may also include both the internal storage unit of the computer device 30 and its external storage device. In this embodiment, the memory 301 is generally used to store the operating system and various application software installed on the computer device 30 . In addition, the memory 301 can also be used to temporarily store various types of data that have been output or will be output.

[0134] The processor 302 may be a central processing unit (Central Processing Unit, CPU), a controller, a microcontroller, a microprocessor, or other data processing chips in some embodiments. The processor 302 is typically used to control the overall operation of the computer device 30 .

[0135] Specifically, in this embodiment, the processor 302 is configured to execute the program of the method for training a model stored in the memory 301, and the program of the method for training a model implements the following steps when executed:

[0136] When a model training instruction is received, attribute information is parsed from the model training instruction, and an algorithm to be trained corresponding to the attribute information is obtained from a preset model library;

[0137] Obtain resource information corresponding to the attribute information from a preset training configuration table, and obtain a training file for training the algorithm to be trained through a preset upload interface;

[0138] Use the resource information and train the algorithm to be trained through the training file to obtain a training result file;

[0139] adding the training result file to a preset model loading component to instantiate the target model;

[0140] A configuration file in the model loading component is modified to deploy the target model into a model consuming object.

[0141] For the specific embodiment process of the above method steps, reference may be made to Embodiment 1, which will not be repeated in this embodiment.