Model production method, device and equipment and storage medium
By recording and optimizing and adjusting the production factors and results in each iteration cycle during the model training process, the problem of inefficient model production in the existing technology is solved, and a more efficient model production process is achieved.
Patent Information
- Application Number
- CN202311478198.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-11-07
- Publication Date
- 2025-05-13
AI Technical Summary
Model production is inefficient in the prior art, especially when multiple model iterations are involved.
By obtaining parameter files, the preset model to be trained based on these parameter files, the production factors and results are recorded, and optimization and adjustments are made based on whether the preset training accuracy conditions are met until the target model file is reached.
It shortens the iteration cycle of model training, improves the efficiency of model production, and makes the model production process more efficient, reproducible and traceable.
Smart Images
Figure CN119988959A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of computer technology, and in particular to a model production method, device, equipment and storage medium. Background Art
[0002] At present, with the development of deep learning algorithms, models trained based on deep learning algorithms are applied to various industries, which increases the demand for deep learning neural network models. Therefore, how to efficiently produce deep learning neural network models is the current development direction in the field.
[0003] Related technologies have proposed a model production method based on a directed acyclic graph, but this method is mainly aimed at one-time model production and is not suitable for model production involving multiple model iterations, resulting in low efficiency of model production. Summary of the invention
[0004] The main purpose of this application is to provide a model production method, device, equipment and storage medium, aiming to solve the technical problem of low efficiency of model production in the prior art.
[0005] To achieve the above objectives, the present application provides a model production method, which comprises:
[0006] S10, obtaining a parameter file;
[0007] S20, based on the parameter file, training the preset model to be trained to obtain production factors and results, wherein the production factors are factors affecting the model output recorded during the model training process, and the results include an updated training model obtained after the model training;
[0008] S30, determining whether the updated training model meets a preset training accuracy condition;
[0009] S40, if the updated training model does not meet the preset training accuracy conditions, then based on the production factors and results, the parameter file is optimized and adjusted to obtain the adjusted parameter file, and the above steps S20-S40 are returned and iterated repeatedly until the updated training model meets the preset training accuracy conditions, and the target model file is obtained, wherein the target model file is used to characterize the correlation between the image to be identified and the target recognition result.
[0010] Optionally, the production factors include a version number of the tool and a version number of the training set, and the output includes an updated training model file and a training log file.
[0011] Optionally, the step of training a preset model to be trained based on the parameter file to obtain production factors and outcomes includes:
[0012] Based on the parameter file, determine the initial record dictionary of production factors and achievements in the current training cycle;
[0013] Based on the parameter file, the preset model to be trained is trained, and the production factors and achievements in the model training process are recorded to obtain the target record dictionary of the updated model, wherein the model initial record dictionary and the target record dictionary include field identifiers and field values.
[0014] Optionally, the step of optimizing and adjusting the parameter file based on the production factors and the achievements to obtain the adjusted parameter file includes:
[0015] Obtain a first preset number of historical model files of the current training model;
[0016] Based on the production factors and achievements corresponding to each of the historical model files, the historical model files are screened for optimal models to obtain a second preset number of screened historical model files;
[0017] Performing sample mining on the filtered historical model files to obtain target samples;
[0018] The target sample is expanded into the parameter file to obtain an adjusted parameter file.
[0019] Optionally, the step of screening the historical model files for optimal models based on the production factors and achievements corresponding to each of the historical model files to obtain a second preset number of screened historical model files includes:
[0020] Based on a preset evaluation function, index mapping is performed on the production factors and achievements corresponding to each of the historical model files to obtain a first evaluation index corresponding to each of the historical model files;
[0021] Aligning the first evaluation indicators corresponding to the historical model files to obtain the aligned first evaluation indicators of the historical model files;
[0022] Calculating the harmonic mean of the aligned first evaluation indexes of each of the historical model files to obtain the evaluation index value of each of the historical model files;
[0023] Based on the evaluation index value of each of the historical model files, each of the historical model files is sorted by performance in descending order to obtain a performance sorting result, and a second preset number of historical model files with the largest evaluation index value in the performance sorting result are selected as the screened historical model files.
[0024] Optionally, the step of performing sample mining on the filtered historical model file to obtain a target sample includes:
[0025] Obtaining a training sample set from a mining sample pool, wherein the training sample set includes training samples newly added for current model training;
[0026] Inputting the training sample set into each filtered historical model file respectively to obtain the prediction result corresponding to each filtered historical model file;
[0027] Based on the prediction result, the training sample set is screened to obtain target samples.
[0028] Optionally, the step of screening the training sample set based on the prediction result to obtain a target sample includes:
[0029] Based on a preset evaluation function, the prediction result is index-mapped to obtain a second evaluation index corresponding to each training sample in the training sample set;
[0030] Determining whether the second evaluation indicator is greater than a preset indicator threshold;
[0031] The training samples corresponding to the second evaluation index greater than the index threshold in the training sample set are used as target samples.
[0032] The present application also provides a model production device, characterized in that the model production device comprises:
[0033] Acquisition module, used to obtain parameter files;
[0034] A training module, used to train the preset model to be trained based on the parameter file to obtain production factors and results, wherein the production factors are factors affecting the model output recorded during the model training process, and the results include an updated training model obtained after the model training;
[0035] A judgment module, used to judge whether the updated training model meets the preset training accuracy condition;
[0036] The iteration module is used to optimize and adjust the parameter file based on the production factors and results if the updated training model does not meet the preset training accuracy conditions, obtain the adjusted parameter file, and return to and iterate the above steps S20-S40 repeatedly until the updated training model meets the preset training accuracy conditions to obtain the target model file, wherein the target model file is used to characterize the correlation between the image to be recognized and the target recognition result.
[0037] And / or, the training module includes: an initial record dictionary determination module, used to determine the initial record dictionary of production factors and achievements in the current training cycle based on the parameter file; a recording module, used to train the preset model to be trained based on the parameter file, and record the production factors and achievements in the model training process to obtain the target record dictionary of the updated model, wherein the model initial record dictionary and the target record dictionary include field identifiers and field values;
[0038] And / or, the iteration module includes: a historical model file acquisition module, used to acquire a first preset number of historical model files of the current training model; a screening module, used to screen the historical model files for optimal models based on the production factors and achievements corresponding to each of the historical model files, to obtain a second preset number of screened historical model files; a mining module, used to perform sample mining on the screened historical model files to obtain target samples; an expansion module, used to expand the target samples to the parameter files to obtain an adjusted parameter file;
[0039] And / or, the screening module includes: a first evaluation module, which is used to perform indicator mapping on the production factors and achievements corresponding to each of the historical model files based on a preset evaluation function to obtain the first evaluation indicators corresponding to each of the historical model files; an indicator alignment module, which is used to perform indicator alignment on the first evaluation indicators corresponding to each of the historical model files to obtain the aligned first evaluation indicators of each of the historical model files; a calculation module, which is used to calculate the harmonic mean of the aligned first evaluation indicators of each of the historical model files to obtain the evaluation indicator value of each of the historical model files; a model screening module, which is used to sort the performance of each of the historical model files in a descending order based on the evaluation indicator value of each of the historical model files to obtain a performance sorting result, and select a second preset number of historical model files with the largest evaluation indicator value in the performance sorting result as the screened historical model files;
[0040] And / or, the mining module includes: a mining sample pool acquisition module, used to obtain a training sample set under the mining sample pool, wherein the training sample set includes training samples newly added for current model training; a prediction module, used to input the training sample set into each filtered historical model file respectively to obtain a prediction result corresponding to each filtered historical model file; a sample screening module, used to screen the training sample set based on the prediction result to obtain a target sample;
[0041] And / or, the sample screening module includes: a second evaluation module, used to perform indicator mapping on the prediction result based on a preset evaluation function to obtain a second evaluation indicator corresponding to each training sample in the training sample set; a threshold judgment module, used to judge whether the second evaluation indicator is greater than a preset indicator threshold; a high-value sample screening module, used to take the training samples corresponding to the second evaluation indicators in the training sample set that are greater than the indicator threshold as target samples.
[0042] The present application also provides a model production device, the model production device comprising: a memory, a processor, and a program stored in the memory for implementing the model production method.
[0043] The memory is used to store a program for implementing the model production method;
[0044] The processor is used to execute a program for implementing the model production method to implement the steps of the model production method.
[0045] The present application also provides a storage medium, on which is stored a program for implementing the model production method, and the program for implementing the model production method is executed by a processor to implement the steps of the model production method.
[0046] This application records and saves the production factors and results of each iteration cycle in the model training process, and based on the production factors and results, optimizes and adjusts the parameter files of the model input to shorten the iteration cycle of model training and improve the efficiency of model production. BRIEF DESCRIPTION OF THE DRAWINGS
[0047] The drawings herein are incorporated into the specification and constitute a part of the specification, illustrate embodiments consistent with the present application, and are used together with the specification to explain the principles of the present application. In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings required for use in the embodiments or the prior art description will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.
[0048] Figure 1 This is a schematic diagram of the process of the first embodiment of the model production method of the present application;
[0049] Figure 2 This is a schematic diagram of the process of the second embodiment of the model production method of the present application;
[0050] Figure 3 This is a module interaction diagram of the second embodiment of the model production method of the present application;
[0051] Figure 4A schematic diagram of the process of model selection for the model production method of the present application;
[0052] Figure 5 A schematic diagram of the target sample mining process of the model production method of this application;
[0053] Figure 6 This is a schematic diagram of the module of the model production device of this application;
[0054] Figure 7 It is a schematic diagram of the device structure of the hardware operating environment involved in the embodiment of the present application.
[0055] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION
[0056] It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.
[0057] Reference Figure 1 , Figure 1 This is a schematic diagram of the process of the first embodiment of the model production method of the present application.
[0058] In a first embodiment, the model production method comprises the following steps:
[0059] S10, obtaining a parameter file;
[0060] It should be noted that the executor of the model production method is a model production device. Preferably, the model production device is an end-to-end system, or it can be other terminals or servers with data transmission and data processing functions, and no specific restrictions are made here.
[0061] Furthermore, the end-to-end system means that the input of the system is its necessary parameters (such as the network structure and data set of the specified model), and the output is the model file in deployment form; the end-to-end form includes four major links: data processing, model training, model indicator evaluation, and model deployment. Compared with the manual processing and connection of each link, the end-to-end form further improves the efficiency of model production.
[0062] It is understandable that the parameter file is a file containing information related to the model to be produced according to user needs, including but not limited to user demand information, model parameter configuration file and web page information, wherein the user demand information includes information related to the model to be produced according to demand, for example, the model corresponding to the user demand business is an OCR text recognition model; the model parameter configuration file includes but is not limited to sample sets such as training sets and test sets used for model training.
[0063] In a specific implementation, the device may obtain the parameter file by extracting the relevant parameter file from the database, or by receiving the parameter file uploaded by the user, which is not specifically limited here.
[0064] S20, based on the parameter file, training the preset model to be trained to obtain production factors and results, wherein the production factors are factors affecting the model output recorded during the model training process, and the results include an updated training model obtained after the model training;
[0065] It should be noted that the process in which the device trains the preset model to be trained based on the parameter file refers to the training of a single iteration cycle, which can be any iteration process under the complete model iteration cycle. The device records the elements in the model training process and the various results produced by the training to obtain the production factors and results produced in the model training process, wherein the production factors are the factors affecting the model output recorded during the model training process, and the results include the updated training model obtained after the model training. This application records and saves the production factors and results in each iteration cycle during the model training process to achieve the reproduction, traceability, order and standardization of model production, which is suitable for model production in various businesses, thereby improving the efficiency of model production.
[0066] In a specific implementation, the production factors include the version number of the tool and the version number of the training set, and the results include the updated training model file and the training log file.
[0067] In a specific implementation, the device trains a preset model to be trained based on the parameter file, and the method for obtaining production factors and achievements also includes:
[0068] The device determines the initial record dictionary of production factors and achievements in the current training cycle based on the parameter file; based on the parameter file, trains the preset model to be trained, and records the production factors and achievements in the model training process to obtain the target record dictionary of the updated model, wherein the model initial record dictionary and the target record dictionary include field identifiers and field values.
[0069] It should be noted that the dictionary record in this application means that after the dictionary is created, the key-value is continuously added. After all records are completed, they are uniformly stored in the database. The device needs to record and save the production factors and results of each iteration cycle in the model training process to the storage database.
[0070] As an example, the complete dictionary recording process includes: 1. In the initial state of the device startup, the device obtains the dictionary D of all records; 2. The device creates an id to identify the uniqueness of this record, where the id is the key, and creates a temporary dictionary D under the model business model ; 3. Record the tool link version, including: Create a tool version id, which is used to identify the uniqueness of the record of this tool environment, recorded as id tool , and create a temporary dictionary D tool , install and obtain the image version of the current system training, with image_version as key and the actual version number as value, and record it in the dictionary D tool ; Secondly, the device records the tool version used by the current system, such as pytorch version, python version, etc., with pytorch_version and python_version as keys and the actual version number as value, and records them into the dictionary D tool ; Finally, the device will be this id tool is the key, dictionary D tool For value, enter the dictionary D model ; 4. The device records the path of this data set, such as training set, test set, etc., specifically including: creating a data set path id, which is used to identify the uniqueness of the record of this data set path, recorded as id data , and create a temporary dictionary D data ; Secondly, the device obtains the current training set, with train_data as the key and the actual path as the value, and records it into the dictionary D data ; The device can also obtain the current verification set, with val_data as the key and the actual path as the value, and record it in the dictionary D data ; The device can also obtain the current original data set, using ori_data as the key and the actual path as the value, and record it in the dictionary D data ; The device can also obtain the current test set, with test_data as the key and the actual path as the value, and record it in the dictionary D data ; After the above records are completed, the device will finally data is the key, dictionary D data For value, enter the dictionary D model ; 5. The device records the pre-trained model used in this training production, with parent_model as the key and the actual pre-trained model as the value, and enters it into the dictionary D model ; 6. The device records the configuration file used in this training, with configs_file as the key and the actual training configuration file as the value, and enters it into the dictionary D model; 7. After the training production of the current iteration cycle of the model is completed, the generated model file is recorded, with model_path as the key and the actual training model as the value, and entered into the dictionary D model ; 8. The log file generated by the device record is entered into the dictionary D with log_path as the key and the actual training model as the value model ; 9. The device records the performance indicators of each test set on this model. The specific methods include: the device creates the id of the performance indicator, which is used to identify the uniqueness of the record of the performance indicator of this model, and records the id eval ; Create a temporary dictionary D eval ; Secondly, the device traverses the test set, using the test set name as the key and the evaluation index as the value, and enters the dictionary D eval ; The device finally sets the id eval is the key, dictionary D eval For value, enter the dictionary D model ; 10. After the above dictionary record is completed, the id is used as the key, and the dictionary D model The actual value is entered into dictionary D to complete the record of all production factors and results.
[0071] S30, determining whether the updated training model meets a preset training accuracy condition;
[0072] In a specific implementation, the device determines whether the updated training model meets the preset training accuracy conditions by inputting samples of the test set into the updated training model, performing error calculation based on the prediction results output by the updated training model and the manually labeled detection results, and determining whether the updated training model meets the preset training accuracy conditions based on the error results; it may also determine whether the updated training model meets the preset training accuracy conditions based on a preset loss function, which is not specifically limited here.
[0073] S40, if the updated training model does not meet the preset training accuracy conditions, then based on the production factors and results, the parameter file is optimized and adjusted to obtain the adjusted parameter file, and the above steps S20-S40 are returned and iterated repeatedly until the updated training model meets the preset training accuracy conditions, and the target model file is obtained, wherein the target model file is used to characterize the correlation between the image to be identified and the target recognition result.
[0074] It should be noted that when the updated training model does not meet the preset training accuracy conditions in the current iteration cycle, the present application proposes that the number of optimization adjustments can be adjusted according to factors such as model size, number of iterations or user needs. For example, the device defaults to the setting that each iteration cycle of model training needs to optimize and adjust the parameter file according to production factors and results before conducting model training for the next iteration cycle. The device modifies the default number of optimization adjustments to optimize and adjust the parameter file after every 5 iteration cycles based on factors such as model size, number of iterations or user needs. The purpose of this setting is to flexibly adjust the production of the model to avoid excessive computational effort of the device.
[0075] It is understandable that the model production method of the present application is not only applicable to short-term one-time task production, but also to multiple periodic task production.
[0076] This application records and saves the production factors and results of each iteration cycle in the model training process, and based on the production factors and results, optimizes and adjusts the parameter files of the model input to shorten the iteration cycle of model training and improve the efficiency of model production.
[0077] Based on the above first embodiment, the present application also provides another embodiment, wherein the model production method comprises:
[0078] In the specific implementation, refer to Figure 2 and Figure 3 The device optimizes and adjusts the parameter file based on the production factors and the results, and the method for obtaining the adjusted parameter file also includes the following steps:
[0079] Step A10, obtaining a first preset number of historical model files of the current training model;
[0080] It should be noted that, during the training of a model service, since the device records all elements of the model training process and various results produced by the training, the historical model files of the model service increase with the increase of the number of iterations. The device can select the number of historical model files according to factors such as the iteration cycle and user needs.
[0081] It is understandable that the device can determine the historical model file by: 1. obtaining the name of the current service to be searched; 2. obtaining the number of target models to be searched, set to N, that is, searching for the N models with the best performance in the historical models; 3. obtaining the generation time range of the model, that is, limiting the search model range; models outside the time range are ignored and not processed. The historical model file is determined in the above manner.
[0082] In a specific implementation, the device obtains a first preset number of historical model files of the current training model from a record database of the model.
[0083] Step A20, based on the production factors and achievements corresponding to each of the historical model files, the historical model files are screened for optimal models to obtain a second preset number of screened historical model files;
[0084] It should be noted that the device screens the historical model files for the optimal model based on the production factors and achievements corresponding to each of the historical model files, and obtains a second preset number of screened historical model files, that is, searching for the top N (i.e., the N historical models with the highest performance ranking) models in the historical models (such as the top 1 model as the pre-trained model for the next iteration, and the top N models for the model-based difficult example mining process), to generate favorable factors for model training in the next iteration cycle, thereby promoting the improvement of model training performance.
[0085] In the specific implementation, refer to Figure 4 The device screens the historical model files for the best model based on the production factors and achievements corresponding to each of the historical model files, and the method for obtaining a second preset number of screened historical model files also includes the following steps:
[0086] Based on a preset evaluation function, the production factors and outputs corresponding to each of the historical model files are mapped to indicators to obtain the first evaluation indicators corresponding to each of the historical model files; the first evaluation indicators corresponding to each of the historical model files are aligned to obtain the aligned first evaluation indicators of each of the historical model files; the harmonic mean of the aligned first evaluation indicators of each of the historical model files is calculated to obtain the evaluation indicator value of each of the historical model files; based on the evaluation indicator value of each of the historical model files, the historical model files are sorted by performance in descending order to obtain a performance sorting result, and a second preset number of historical model files with the largest evaluation indicator value in the performance sorting result are selected as the screened historical model files.
[0087] As an example: when the category of the model to be trained is a detection model, the indicator evaluation method of the evaluation function of the detection model includes: (1) calculating the iou of the model prediction box and the GT box (the accurate target box manually annotated) in an image; (2) when the iou is greater than the set threshold (such as 0.5), the prediction box and the GT box are considered to be in a matching state, and the number of GT boxes that match at this time is counted; (3) the number of all prediction boxes and GT boxes in the image at this time is counted; (4) the statistical range is expanded to all test set images, and the number of GT boxes that match at this time is calculated, recorded as match_num, and the number of all GT boxes is recorded as total_gt_num, and the number of all prediction boxes is recorded as total_gt_num. The number is denoted as total_pred_num; (5) Calculate the first evaluation indicator of the detection evaluation function, recall rate recall, recall = match_num / total_gt_num; (6) The second evaluation indicator precision, precision = match_num / total_pred_num; (7) The third evaluation indicator f1-score, the harmonic mean, takes into account both precision and recall, and its expression is as follows f1-score = 2*precision*recall / (precision+recall).
[0088] Furthermore, when the category of the model to be trained is a recognition model, the indicator evaluation method of the evaluation function of the recognition model includes: (1) calculating whether the model prediction string is identical to the GT string. If they are identical, the prediction is correct; if they are different, the prediction is incorrect. (2) expanding the statistical scope to all test sets, calculating the number of correct predictions divided by the total number of GT strings, and this indicator is referred to as accuracy.
[0089] In the specific implementation, the device needs to determine whether the first evaluation indicators corresponding to each of the historical model files are aligned, such as whether the versions of the deep learning framework pytorch and the inference library are aligned. If they cannot be aligned, the indicators of the remaining historical models will be recalculated based on the environment on which the latest model depends.
[0090] Specifically, the device first determines whether the test sets of each of the historical model files are aligned. For example, the A model has two test sets a and b, and the latest C model has four test sets a', b, e, and f. It can be seen that the test sets of the A and C models cannot be aligned. Specifically, C has four test sets, which are two more than the A model, e and f. At the same time, C has an a' test set, while A has an a test set. The difference is that due to the iterative update of the version, the data set a has also been modified and updated, and the updated test set is recorded as a'. Therefore, if the test sets cannot be aligned, the test set corresponding to the latest model shall prevail, and the indicators of the remaining historical models shall be recalculated.
[0091] Secondly, based on the above alignment judgment results, if the device determines that there are factors that cannot be aligned, it is necessary to recalculate the indicators corresponding to the remaining historical models based on the environment on which the latest model depends and its corresponding test set.
[0092] Furthermore, after the device obtains the first evaluation index after alignment of each of the historical model files, since each model has m test sets (m is an integer greater than 0), the harmonic mean is calculated for the indicators on the m test sets; the harmonic mean is greatly affected by the minimum value, so this reflects that the model has better indicators for the m test sets, so this application chooses to use the harmonic mean indicator value conversion algorithm, where the expression of the final indicator of each model is: score = n / (1 / score1+1 / score2+…+1 / scorem).
[0093] Finally, based on the evaluation index values of each of the historical model files, each of the historical model files is sorted by performance in descending order to obtain a performance sorting result, and a second preset number of historical model files with the largest evaluation index values in the performance sorting result are selected as the screened historical model files.
[0094] For example, the evaluation index value of historical model file A is 3, the evaluation index value of historical model file B is 4, the evaluation index value of historical model file C is 2, and the evaluation index value of historical model file D is 5. The performance ranking result is historical model file D-historical model file B-historical model file A-historical model file C, where the second preset number is three, and the screened historical model files are historical model file D, historical model file B, and historical model file A.
[0095] It should be noted that the second preset number is at least 3.
[0096] Step A30, performing sample mining on the screened historical model file to obtain a target sample;
[0097] It should be noted that the target samples refer to high-value training samples, which can also be understood as difficult examples that cannot be accurately identified by the model currently being trained. The device obtains the topN models (i.e., the historical model files after the second preset number of screening), and then mines the difficult example data, and then uses the mined data as the training set produced by this model training, which helps to improve the performance of the model.
[0098] In a specific implementation, the device performs sample mining on the filtered historical model file, and the method for obtaining the target sample further includes the following steps:
[0099] Obtain a training sample set under the mining sample pool, wherein the training sample set includes newly added training samples for current model training; input the training sample set into each filtered historical model file respectively to obtain the prediction results corresponding to each filtered historical model file; based on the prediction results, perform sample screening on the training sample set to obtain the target sample.
[0100] It should be noted that the mining sample pool contains multiple image sample sets to be mined. The device mines high-value samples and expands the high-value samples into the training sample set of the model to improve the prediction accuracy of the model.
[0101] In a specific implementation, the device performs sample screening on the training sample set based on the prediction result, and the method for obtaining the target sample further includes the following steps:
[0102] Based on a preset evaluation function, the prediction result is index mapped to obtain a second evaluation index corresponding to each training sample in the training sample set; whether the second evaluation index is greater than a preset index threshold is determined; and the training sample corresponding to the second evaluation index greater than the index threshold in the training sample set is used as the target sample.
[0103] As an example, see Figure 5 The device performs sample mining on the filtered historical model files, and the method for obtaining the target sample specifically includes: 1. Obtaining L training sample sets under the mining sample pool; 2. Traversing the current L training sets, specifically, inputting the training sample sets into N filtered historical model files respectively, obtaining the prediction results corresponding to each filtered historical model file, that is, obtaining the inference results of N models; for each sample, assuming that the prediction result corresponding to each model is j n , and for each sample, let the GT (manually labeled target detection box) of the sample be k, and calculate the j according to the evaluation function nThe prediction result and the evaluation index of GT, wherein the calculation method of the evaluation index refers to the mapping of the evaluation index under the above optimal model screening; when the evaluation index of the sample is greater than a threshold value (such as 0.5), it is determined to be a simple example sample, and if it is less than the threshold value, it is determined to be a difficult example sample; after the device obtains the difficult example samples of N models, if the proportion of difficult examples is greater than 50%, the current sample is determined to be a difficult example sample. For example, the screened historical model file includes model A, model B and model C, and the samples determined to be difficult examples by model A include sample X, sample Y and sample Z, and the samples determined to be difficult examples by model B include sample X and sample Z, and the samples determined to be difficult examples by model C include sample Z, wherein the proportion of sample X and sample Z is greater than 50%, and the proportion of sample Y is less than 50%, so the target samples are sample X and sample Z. Finally, the device expands sample X and sample Z to the parameter file, uses it as a training set, and supplements it into this model training production.
[0104] Step A40, expanding the target sample into the parameter file to obtain an adjusted parameter file.
[0105] It is understandable that the present application proposes to record and save the production factors and results of each iteration cycle in the model training process, that is, to build a model factor management module based on the database, and use this module to record the various factors and results in the model training process to ensure the reproducibility and traceability of the experiment. At the same time, this is also the basis for the subsequent model selection module.
[0106] The present application further proposes a model optimization selector, through which the topN performance models can be effectively found and selected, and these advanced models are used to generate favorable factors for the next iteration; for example, the top1 model is used as the pre-training model for the next iteration, and the topN models are used in the model-based difficult example mining process.
[0107] Based on the above two points, this solution has effectively promoted the closed-loop and rapid iteration of model training, and promoted orderly and standardized model training production.
[0108] The present application also provides a model production device, referring to Figure 6 , the model production device comprises:
[0109] An acquisition module 10 is used to acquire a parameter file;
[0110] The training module 20 is used to train the preset model to be trained based on the parameter file to obtain production factors and results, wherein the production factors are factors affecting the model output recorded during the model training process, and the results include an updated training model obtained after the model training;
[0111] A judging module 30 is used to judge whether the updated training model meets the preset training accuracy condition;
[0112] The iteration module 40 is used to optimize and adjust the parameter file based on the production factors and results if the updated training model does not meet the preset training accuracy conditions, obtain the adjusted parameter file, and return and repeat the above steps S20-S40 until the updated training model meets the preset training accuracy conditions to obtain the target model file, wherein the target model file is used to characterize the correlation between the image to be recognized and the target recognition result.
[0113] And / or, the training module 20 includes: an initial record dictionary determination module, which is used to determine the initial record dictionary of production factors and achievements in the current training cycle based on the parameter file; a recording module, which is used to train the preset model to be trained based on the parameter file, and record the production factors and achievements in the model training process to obtain the target record dictionary of the updated model, wherein the model initial record dictionary and the target record dictionary include field identifiers and field values;
[0114] And / or, the iteration module 40 includes: a historical model file acquisition module, used to acquire a first preset number of historical model files of the current training model; a screening module, used to screen the historical model files for optimal models based on the production factors and achievements corresponding to each of the historical model files, to obtain a second preset number of screened historical model files; a mining module, used to perform sample mining on the screened historical model files to obtain target samples; an expansion module, used to expand the target samples to the parameter files to obtain an adjusted parameter file;
[0115] And / or, the screening module includes: a first evaluation module, which is used to perform indicator mapping on the production factors and achievements corresponding to each of the historical model files based on a preset evaluation function to obtain the first evaluation indicators corresponding to each of the historical model files; an indicator alignment module, which is used to perform indicator alignment on the first evaluation indicators corresponding to each of the historical model files to obtain the aligned first evaluation indicators of each of the historical model files; a calculation module, which is used to calculate the harmonic mean of the aligned first evaluation indicators of each of the historical model files to obtain the evaluation indicator value of each of the historical model files; a model screening module, which is used to sort the performance of each of the historical model files in a descending order based on the evaluation indicator value of each of the historical model files to obtain a performance sorting result, and select a second preset number of historical model files with the largest evaluation indicator value in the performance sorting result as the screened historical model files;
[0116] And / or, the mining module includes: a mining sample pool acquisition module, used to obtain a training sample set under the mining sample pool, wherein the training sample set includes training samples newly added for current model training; a prediction module, used to input the training sample set into each filtered historical model file respectively to obtain a prediction result corresponding to each filtered historical model file; a sample screening module, used to screen the training sample set based on the prediction result to obtain a target sample;
[0117] And / or, the sample screening module includes: a second evaluation module, used to perform indicator mapping on the prediction result based on a preset evaluation function to obtain a second evaluation indicator corresponding to each training sample in the training sample set; a threshold judgment module, used to judge whether the second evaluation indicator is greater than a preset indicator threshold; a high-value sample screening module, used to take the training samples corresponding to the second evaluation indicators in the training sample set that are greater than the indicator threshold as target samples.
[0118] The specific implementation of the model production device of the present application is basically the same as the embodiments of the above-mentioned model production method, and will not be repeated here.
[0119] Reference Figure 7 , Figure 7 It is a schematic diagram of the terminal structure of the hardware operating environment involved in the embodiment of the present application.
[0120] like Figure 7 As shown, the terminal may include: a processor 1001, such as a CPU, a network interface 1004, a user interface 1003, a memory 1005, and a communication bus 1002. Among them, the communication bus 1002 is used to realize the connection and communication between these components. The user interface 1003 may include a display screen (Display), an input unit such as a keyboard (Keyboard), and the optional user interface 1003 may also include a standard wired interface and a wireless interface. The network interface 1004 may optionally include a standard wired interface and a wireless interface (such as a WI-FI interface). The memory 1005 may be a high-speed RAM memory, or a stable memory (non-volatile memory), such as a disk memory. The memory 1005 may also be a storage device independent of the aforementioned processor 1001.
[0121] Optionally, the model production device may also include a rectangular user interface, a network interface, a camera, an RF (Radio Frequency) circuit, a sensor, an audio circuit, a WiFi module, etc. The rectangular user interface may include a display screen (Display), an input submodule such as a keyboard (Keyboard), and the optional rectangular user interface may also include a standard wired interface and a wireless interface. The network interface may optionally include a standard wired interface and a wireless interface (such as a WI-FI interface).
[0122] Those skilled in the art will understand that Figure 6 The model production equipment structure shown in the figure does not constitute a limitation on the model production equipment, and may include more or less components than shown in the figure, or combine certain components, or arrange the components differently.
[0123] like Figure 7 As shown, the memory 1005 as a storage medium may include an operating system, a network communication module and a model production program. The operating system is a program that manages and controls the hardware and software resources of the model production device, and supports the operation of the model production program and other software and / or programs. The network communication module is used to realize the communication between the components inside the memory 1005, and the communication with other hardware and software in the model production system.
[0124] exist Figure 7 In the model production device shown, the processor 1001 is used to execute the model production program stored in the memory 1005 to implement the steps of any of the above-mentioned model production methods.
[0125] The specific implementation of the model production equipment of the present application is basically the same as the embodiments of the above-mentioned model production method, and will not be repeated here.
[0126] The present application also provides a storage medium, on which is stored a program for implementing the model production method, and the program for implementing the model production method is executed by a processor to implement the model production method as described below:
[0127] S10, obtaining a parameter file;
[0128] S20, based on the parameter file, training the preset model to be trained to obtain production factors and results, wherein the production factors are factors affecting the model output recorded during the model training process, and the results include an updated training model obtained after the model training;
[0129] S30, determining whether the updated training model meets a preset training accuracy condition;
[0130] S40, if the updated training model does not meet the preset training accuracy conditions, then based on the production factors and results, the parameter file is optimized and adjusted to obtain the adjusted parameter file, and the above steps S20-S40 are returned and iterated repeatedly until the updated training model meets the preset training accuracy conditions, and the target model file is obtained, wherein the target model file is used to characterize the correlation between the image to be identified and the target recognition result.
[0131] Optionally, the production factors include a version number of the tool and a version number of the training set, and the output includes an updated training model file and a training log file.
[0132] Optionally, the step of training a preset model to be trained based on the parameter file to obtain production factors and outcomes includes:
[0133] Based on the parameter file, determine the initial record dictionary of production factors and achievements in the current training cycle;
[0134] Based on the parameter file, the preset model to be trained is trained, and the production factors and achievements in the model training process are recorded to obtain the target record dictionary of the updated model, wherein the model initial record dictionary and the target record dictionary include field identifiers and field values.
[0135] Optionally, the step of optimizing and adjusting the parameter file based on the production factors and the achievements to obtain the adjusted parameter file includes:
[0136] Obtain a first preset number of historical model files of the current training model;
[0137] Based on the production factors and achievements corresponding to each of the historical model files, the historical model files are screened for optimal models to obtain a second preset number of screened historical model files;
[0138] Performing sample mining on the filtered historical model files to obtain target samples;
[0139] The target sample is expanded into the parameter file to obtain an adjusted parameter file.
[0140] Optionally, the step of screening the historical model files for optimal models based on the production factors and achievements corresponding to each of the historical model files to obtain a second preset number of screened historical model files includes:
[0141] Based on a preset evaluation function, index mapping is performed on the production factors and achievements corresponding to each of the historical model files to obtain a first evaluation index corresponding to each of the historical model files;
[0142] Aligning the first evaluation indicators corresponding to the historical model files to obtain the aligned first evaluation indicators of the historical model files;
[0143] Calculating the harmonic mean of the aligned first evaluation indexes of each of the historical model files to obtain the evaluation index value of each of the historical model files;
[0144] Based on the evaluation index value of each of the historical model files, each of the historical model files is sorted by performance in descending order to obtain a performance sorting result, and a second preset number of historical model files with the largest evaluation index value in the performance sorting result are selected as the screened historical model files.
[0145] Optionally, the step of performing sample mining on the filtered historical model file to obtain a target sample includes:
[0146] Obtaining a training sample set from a mining sample pool, wherein the training sample set includes training samples newly added for current model training;
[0147] Inputting the training sample set into each filtered historical model file respectively to obtain the prediction result corresponding to each filtered historical model file;
[0148] Based on the prediction result, the training sample set is screened to obtain target samples.
[0149] Optionally, the step of screening the training sample set based on the prediction result to obtain a target sample includes:
[0150] Based on a preset evaluation function, the prediction result is index-mapped to obtain a second evaluation index corresponding to each training sample in the training sample set;
[0151] Determining whether the second evaluation indicator is greater than a preset indicator threshold;
[0152] The training samples corresponding to the second evaluation index greater than the index threshold in the training sample set are used as target samples.
[0153] The specific implementation method of the storage medium of the present application is basically the same as the embodiments of the above-mentioned model production method, and will not be repeated here.
[0154] The present application also provides a computer program product, including a computer program, which implements the steps of the above-mentioned model production method when executed by a processor.
[0155] The specific implementation of the computer program product of the present application is basically the same as the embodiments of the above-mentioned model production method, and will not be repeated here.
[0156] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.
[0157] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.
[0158] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present application.
[0159] The above are only preferred embodiments of the present application, and are not intended to limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.
Claims
1. A model production method, characterized in that: The model production method comprises: S10, obtaining a parameter file; S20, based on the parameter file, training the preset model to be trained to obtain production factors and results, wherein the production factors are factors affecting the model output recorded during the model training process, and the results include an updated training model obtained after the model training; S30, determining whether the updated training model meets a preset training accuracy condition; S40, if the updated training model does not meet the preset training accuracy conditions, then based on the production factors and results, the parameter file is optimized and adjusted to obtain the adjusted parameter file, and the above steps S20-S40 are returned and iterated repeatedly until the updated training model meets the preset training accuracy conditions, and the target model file is obtained, wherein the target model file is used to characterize the correlation between the image to be identified and the target recognition result.
2. The model production method according to claim 1, characterized in that: The production factors include the version number of the tool and the version number of the training set, and the results include the updated training model file and the training log file.
3. The model production method according to claim 1, characterized in that: The step of training the preset model to be trained based on the parameter file to obtain production factors and achievements includes: Based on the parameter file, determine the initial record dictionary of production factors and achievements in the current training cycle; Based on the parameter file, the preset model to be trained is trained, and the production factors and achievements in the model training process are recorded to obtain the target record dictionary of the updated model, wherein the model initial record dictionary and the target record dictionary include field identifiers and field values.
4. The model production method according to claim 1, characterized in that: The step of optimizing and adjusting the parameter file based on the production factors and the achievements to obtain the adjusted parameter file comprises: Obtain a first preset number of historical model files of the current training model; Based on the production factors and achievements corresponding to each of the historical model files, the historical model files are screened for optimal models to obtain a second preset number of screened historical model files; Performing sample mining on the filtered historical model files to obtain target samples; The target sample is expanded into the parameter file to obtain an adjusted parameter file.
5. The model production method according to claim 4, characterized in that: The step of screening the historical model files for optimal models based on the production factors and achievements corresponding to each of the historical model files to obtain a second preset number of screened historical model files includes: Based on a preset evaluation function, index mapping is performed on the production factors and achievements corresponding to each of the historical model files to obtain a first evaluation index corresponding to each of the historical model files; Aligning the first evaluation indicators corresponding to the historical model files to obtain the aligned first evaluation indicators of the historical model files; Calculating the harmonic mean of the aligned first evaluation indexes of each of the historical model files to obtain the evaluation index value of each of the historical model files; Based on the evaluation index value of each of the historical model files, each of the historical model files is sorted by performance in descending order to obtain a performance sorting result, and a second preset number of historical model files with the largest evaluation index value in the performance sorting result are selected as the screened historical model files.
6. The model production method according to claim 4, characterized in that: The step of performing sample mining on the filtered historical model file to obtain a target sample includes: Obtaining a training sample set from a mining sample pool, wherein the training sample set includes training samples newly added for current model training; Inputting the training sample set into each filtered historical model file respectively to obtain the prediction result corresponding to each filtered historical model file; Based on the prediction result, the training sample set is screened to obtain target samples.
7. The model production method according to claim 6, characterized in that: The step of screening the training sample set based on the prediction result to obtain a target sample comprises: Based on a preset evaluation function, the prediction result is index-mapped to obtain a second evaluation index corresponding to each training sample in the training sample set; Determining whether the second evaluation indicator is greater than a preset indicator threshold; The training samples corresponding to the second evaluation index greater than the index threshold in the training sample set are used as target samples.
8. A model production device, characterized in that: The model production device comprises: Acquisition module, used to obtain parameter files; A training module, used to train the preset model to be trained based on the parameter file to obtain production factors and results, wherein the production factors are factors affecting the model output recorded during the model training process, and the results include an updated training model obtained after the model training; A judgment module, used to judge whether the updated training model meets the preset training accuracy condition; The iteration module is used to optimize and adjust the parameter file based on the production factors and results if the updated training model does not meet the preset training accuracy conditions, obtain the adjusted parameter file, and return to and iterate the above steps S20-S40 repeatedly until the updated training model meets the preset training accuracy conditions to obtain the target model file, wherein the target model file is used to characterize the correlation between the image to be recognized and the target recognition result.
9. The model production device according to claim 8, characterized in that: The training module comprises: An initial record dictionary determination module is used to determine the initial record dictionary of production factors and achievements in the current training cycle based on the parameter file; a recording module is used to train the preset model to be trained based on the parameter file, and record the production factors and achievements in the model training process to obtain the target record dictionary of the updated model, wherein the model initial record dictionary and the target record dictionary include field identifiers and field values; And / or, the iteration module includes: a historical model file acquisition module, used to acquire a first preset number of historical model files of the current training model; a screening module, used to screen the historical model files for optimal models based on the production factors and achievements corresponding to each of the historical model files, to obtain a second preset number of screened historical model files; a mining module, used to perform sample mining on the screened historical model files to obtain target samples; an expansion module, used to expand the target samples to the parameter files to obtain an adjusted parameter file; And / or, the screening module includes: a first evaluation module, which is used to perform indicator mapping on the production factors and achievements corresponding to each of the historical model files based on a preset evaluation function to obtain the first evaluation indicators corresponding to each of the historical model files; an indicator alignment module, which is used to perform indicator alignment on the first evaluation indicators corresponding to each of the historical model files to obtain the aligned first evaluation indicators of each of the historical model files; a calculation module, which is used to calculate the harmonic mean of the aligned first evaluation indicators of each of the historical model files to obtain the evaluation indicator value of each of the historical model files; a model screening module, which is used to sort the performance of each of the historical model files in a descending order based on the evaluation indicator value of each of the historical model files to obtain a performance sorting result, and select a second preset number of historical model files with the largest evaluation indicator value in the performance sorting result as the screened historical model files; And / or, the mining module includes: a mining sample pool acquisition module, used to obtain a training sample set under the mining sample pool, wherein the training sample set includes training samples newly added for current model training; a prediction module, used to input the training sample set into each filtered historical model file respectively to obtain a prediction result corresponding to each filtered historical model file; a sample screening module, used to screen the training sample set based on the prediction result to obtain a target sample; And / or, the sample screening module includes: a second evaluation module, used to perform indicator mapping on the prediction result based on a preset evaluation function to obtain a second evaluation indicator corresponding to each training sample in the training sample set; a threshold judgment module, used to judge whether the second evaluation indicator is greater than a preset indicator threshold; a high-value sample screening module, used to take the training samples corresponding to the second evaluation indicators in the training sample set that are greater than the indicator threshold as target samples.
10. A model production device, characterized in that: The model production device comprises: a memory, a processor and a program stored in the memory for implementing the model production method. The memory is used to store a program for implementing the model production method; The processor is used to execute a program for implementing the model production method, so as to implement the steps of the model production method according to any one of claims 1 to 7.
11. A storage medium, characterized in that: The storage medium stores a program for implementing the model production method, and the program for implementing the model production method is executed by a processor to implement the steps of the model production method according to any one of claims 1 to 7.