Prediction file release method and system, computer equipment and storage medium

By preprocessing and feature extraction of the history records of the file release system, building prediction models and building automated file release services, it solves the problem that users find it difficult to standardize the issuance of publishing objects and quantities, and realizes efficient file release management and reduces managers workload and audit frequency.

CN119089340APending Publication Date: 2024-12-06BEIJING QIHOOD TECHNOLOGY CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411052738.2
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-01
Publication Date
2024-12-06

AI Technical Summary

Technical Problem

When the existing file release system is launched or updated, it is difficult for users to standardize the release objects and quantities, resulting in QA requiring multiple reviews and modifications, which increases the release workload and the process is cumbersome and time-consuming.

Method used

By preprocessing and feature extraction of the release application history of the file release system, a file release prediction model is constructed, prediction services are built, file release management operations are optimized, and managers are reduced. Specific steps include removing dirty data, filling vacant fields, feature selection, data processing, model training and service construction, and using decision trees and random forest algorithms to achieve automated file release.

Benefits of technology

It realizes the specifications for the release objects and quantity of newly launched or updated documents when submitting the release application, reduces invalid data, reduces the frequency of QA review and modification, improves release efficiency, reduces the workload of managers, and solves the existing documents in the technology. The cumbersome and time-consuming issue of publishing systems.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119089340A_ABST
    Figure CN119089340A_ABST
Patent Text Reader

Abstract

The invention provides a method and system for predicting file release, computer equipment and a storage medium, and the method comprises the steps: carrying out the preprocessing of a release application historical record of a file release system, and obtaining a historical record meeting a preset demand; performing feature extraction on the historical records meeting the preset requirements to obtain a trainable data set; using a preset first algorithm to construct a file release prediction model by using the data set; and building a file release prediction service according to the file release prediction model. Therefore, the file publishing and updating publishing management operation is optimized, and the file publishing workload of a manager is reduced.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a method, system, computer equipment and storage medium for predicting file release. Background Art

[0002] The file release update and new online release management system, QA (QUALITY ASSURANCE), development, product, testing, and approval groups control the file release on this system. In the process of submitting release applications, users often fail to provide standardized information on the objects and quantities of new or updated files. The system has a lot of invalid data, which requires QA to review and modify again, increasing the workload of release. In addition, compared with the existing file release submission process, the file release system release application submission process is that users submit relevant applications on this system and provide the release object, ratio, and grayscale value of the release file. After submission, QA will modify and adjust according to the release requirements, and finally the approval group will approve it. This process involves multiple departments. If the review fails, it needs to be re-communicated and submitted again. This process is time-consuming and cumbersome.

[0003] In summary, the prior art obviously has inconveniences and defects in practical use, so it is necessary to improve it. Summary of the invention

[0004] In view of the above-mentioned defects, the purpose of the present invention is to provide a method, system, computer device and storage medium for predicting file release, so as to optimize the release management operations of files and update them, and reduce the workload of managers in file release.

[0005] In order to achieve the above object, the present invention provides a method for predicting file release, comprising:

[0006] Pre-process the release application history records of the document release system to obtain the history records that meet the preset requirements;

[0007] Extracting features from the historical records that meet the preset requirements to obtain a trainable data set;

[0008] Using a preset first algorithm and the data set to construct a file release prediction model;

[0009] A file release prediction service is established based on the file release prediction model.

[0010] According to the method, the preprocessing of the release application history records of the file release system includes:

[0011] According to the maintenance requirements of the file publishing system, remove the dirty data in the publishing application history record;

[0012] For the vacant fields in the release application history record, find fields related to the vacant fields, use the related fields to predict the content of the vacant fields, and give the vacant fields a default filling value;

[0013] The feature extraction of the historical records meeting the preset requirements includes:

[0014] Removing fields irrelevant to the release of the forecast file from the historical records that meet the preset requirements;

[0015] The units of the numerical fields in the historical records that meet the preset requirements are removed, and the dimensions of the numerical fields are unified and the data is normalized to obtain the first type of numerical fields related to the release of the forecast file;

[0016] Using unique digital identifiers for non-fields of the numerical type in the historical records that meet the preset requirements, to obtain fields of the second type of numerical type related to the release of the forecast file;

[0017] Using a preset second algorithm to evaluate the contribution value of each field in the first category of numerical fields and the second category of numerical fields to the release of the prediction file, and selecting valid fields that are effective for the release of the prediction file according to the contribution values, and marking the valid fields as record features;

[0018] The step of obtaining a trainable data set comprises:

[0019] Complete feature extraction of the historical records that meet the preset requirements, and obtain all valid fields marked as the record features to form the data set;

[0020] The step of using the preset first algorithm to construct a file release prediction model using the data set includes:

[0021] Splitting the data set into a training set and a test set according to a preset ratio;

[0022] Constructing the file release prediction model using a preset first algorithm on the training set;

[0023] Testing the file release prediction model on the test set, and calculating the prediction accuracy of the file release prediction model;

[0024] When the accuracy of the file release prediction model reaches the expected effect, the file release prediction model is the final file release prediction model.

[0025] According to the method, after calculating the prediction accuracy of the file release prediction model, the method further includes: continuously adjusting the relevant parameters of the file release prediction model according to the expected effect;

[0026] The preset first algorithm is a decision tree algorithm; the preset second algorithm is a random forest algorithm; the preset ratio is 8:2;

[0027] The prediction accuracy of the file release prediction model is calculated using the accuracy_score method in the sklearn library;

[0028] The file release prediction model related parameters include the learning rate, the number of iterations and the depth of the tree in the decision tree algorithm.

[0029] According to the method, building a file release prediction service according to the file release prediction model includes:

[0030] Using a preset framework to build the document publishing prediction service, and integrating it into the document publishing system in a preset connection mode;

[0031] When the file publishing prediction service receives a publishing prediction request, it verifies whether the publishing prediction request has been predicted, and if so, directly extracts the prediction result of the file publishing prediction model from the cache of the file publishing system and returns it;

[0032] If not, the release prediction request is added to the task queue of the file release system. After getting the request from the task queue, the data in the release prediction request is parsed, and feature extraction is performed on the data for the historical records that meet the preset requirements. After the trainable data set is processed, it is input into the file release prediction model. After the file release prediction model gives a prediction value, it is returned and added to the cache.

[0033] According to the method, building a file release prediction service according to the file release prediction model also includes:

[0034] Regularly extract new file release records from a preset database, update the file release prediction model online, and optimize the file release prediction model;

[0035] The types of file release prediction models include: file release object prediction model, file release ratio prediction model, and file release gray value prediction model;

[0036] The preset frameworks are Flask framework, Django framework and Tornado framework; the preset connection method is the web interface method.

[0037] In order to achieve another object of the present invention, the present invention further provides a system for predicting file release, comprising:

[0038] A preprocessing module is used to preprocess the release application history records of the file release system to obtain the history records that meet the preset requirements;

[0039] A feature extraction module is used to extract features from the historical records that meet the preset requirements to obtain a trainable data set;

[0040] A model building module, used to use a preset first algorithm to build a file release prediction model using the data set;

[0041] The service building module is used to build a file release prediction service based on the file release prediction model.

[0042] According to the system, the preprocessing module comprises:

[0043] A first removal submodule is used to remove dirty data in the release application history records according to the maintenance requirements of the file release system;

[0044] A gap filling submodule is used to find the fields related to the vacant fields in the release application history record, predict the content of the vacant fields by using the related fields, and fill the vacant fields with default values;

[0045] The feature extraction module comprises:

[0046] A second removal submodule, configured to remove fields irrelevant to the release of the prediction file from the historical records that meet the preset requirements;

[0047] A third removal submodule is used to remove units from the numerical fields in the historical records that meet the preset requirements, and to unify the dimensions and perform data normalization on the numerical fields to obtain the first type of numerical fields related to the release of the forecast file;

[0048] An identification submodule, used for using a unique digital identification for the non-fields of the numerical type in the historical records that meet the preset requirements, to obtain the fields of the second type of numerical type related to the release of the forecast file;

[0049] An evaluation submodule, configured to evaluate the contribution value of each field in the first category of numerical fields and the second category of numerical fields to the release of the prediction file by using a preset second algorithm, and select valid fields that are effective for the release of the prediction file according to the contribution values, and mark the valid fields as record features;

[0050] The feature extraction module completes feature extraction of the historical records that meet the preset requirements, and all valid fields marked as the record features obtained constitute the data set;

[0051] The model building module includes:

[0052] A splitting submodule, used to split the data set into a training set and a test set according to a preset ratio;

[0053] A construction submodule, configured to construct the file release prediction model using a preset first algorithm on the training set;

[0054] A calculation submodule, used for testing the file release prediction model on the test set and calculating the prediction accuracy of the file release prediction model;

[0055] The finalization submodule is used to determine that when the accuracy of the file release prediction model reaches the expected effect, the file release prediction model is the final file release prediction model.

[0056] According to the system, the service building module includes:

[0057] A construction submodule is used to construct the file publishing prediction service using a preset framework and integrate it into the file publishing system in a preset connection mode;

[0058] A service building submodule is used for, when the file publishing prediction service receives a publishing prediction request, verifying whether the publishing prediction request has been predicted, and if so, directly extracting the prediction result of the file publishing prediction model from the cache of the file publishing system and returning it;

[0059] If not, the release prediction request is added to the task queue of the file release system. After getting the request from the task queue, the data in the release prediction request is parsed, and feature extraction is performed on the data for the historical records that meet the preset requirements. After the trainable data set is processed, it is input into the file release prediction model. After the file release prediction model gives a prediction value, it is returned and added to the cache.

[0060] In order to achieve another object of the present invention, the present invention further provides a storage medium for storing a computer program for executing any one of the above-mentioned methods for predicting file release.

[0061] In order to achieve another object of the present invention, the present invention also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method for publishing a forecast document as described in any one of the above when executing the computer program.

[0062] The present invention first preprocesses the release application history records of the file release system, performs feature extraction on the history records that meet the preset requirements, and after obtaining a trainable data set, uses a preset first algorithm to use the data set to build a file release prediction model; finally, a file release prediction service is built according to the file release prediction model, thereby optimizing the release management operations of file release and update, and reducing the workload of managers for file release. Through the provision of the present invention, managers can provide the objects and quantity specifications for the release of new or updated files during the submission of release applications, remove invalid data, and do not need QA to review and modify again, thereby reducing the release workload. The present invention implements an automated file release solution through the GBDT algorithm, which solves the pain points of the file release system in the prior art. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] Figure 1 It is a schematic diagram of the system composition of the prediction file release provided by the embodiment of the present invention;

[0064] Figure 2 It is a schematic diagram of the system composition of the prediction file release provided by the embodiment of the present invention;

[0065] Figure 3 This is a schematic diagram of file release prediction implemented by a system for predicting file release provided by an embodiment of the present invention;

[0066] Figure 4 is a flow chart of a method for predicting file release provided by an embodiment of the present invention;

[0067] Figure 5 is a schematic diagram of the composition of a computer device provided by an embodiment of the present invention;

[0068] Figure 6 It is a schematic diagram of the structure of a computer-readable storage medium provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0069] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0070] It should be noted that references to "one embodiment", "embodiment", "example embodiment", etc. in this specification refer to the embodiment described, which may include specific features, structures or characteristics, but not every embodiment must include these specific features, structures or characteristics. In addition, such expressions do not refer to the same embodiment. Furthermore, when describing specific features, structures or characteristics in conjunction with an embodiment, whether or not there is an explicit description, it has been shown that it is within the knowledge of those skilled in the art to combine such features, structures or characteristics into other embodiments.

[0071] In addition, certain words are used in the specification and subsequent claims to refer to specific components or parts. Those with ordinary knowledge in the relevant field should understand that manufacturers can use different nouns or terms to refer to the same component or part. This specification and subsequent claims do not use differences in names as a way to distinguish components or parts, but use differences in the functions of components or parts as the criteria for distinction. "Including" and "including" mentioned throughout the specification and subsequent claims are open-ended terms and should be interpreted as "including but not limited to". In addition, the word "connected" here includes any direct and indirect electrical connection means. Indirect electrical connection means include connection through other devices.

[0072] See also Figure 1 In one embodiment of the present invention, a system 100 for predicting file release is provided, comprising:

[0073] The preprocessing module 10 is used to preprocess the release application history records of the file release system to obtain the history records that meet the preset requirements; the preprocessing module 10 preprocesses the release application history records of the file release system to make the processed history records useful data for file release; in one embodiment of the present invention, the file release system is a 360 file release system, and the 360 ​​file release system is a 360 Security Guard file update and new online release management system;

[0074] The feature extraction module 20 is used to extract features from the historical records that meet the preset requirements to obtain a trainable data set; the feature extraction module 20 uses relevant feature engineering technology to extract features from the historical records preprocessed by the preprocessing module 10, and the trainable data set obtained thereby serves as the basis of the file release prediction model;

[0075] The model building module 30 is used to use the preset first algorithm to build a file release prediction model using the data set; the model building module 30 builds a file release prediction model based on the preset first algorithm; through the establishment of the model, the user can give the object and quantity specifications of the new online or updated file release;

[0076] The service building module 40 is used to build a file release prediction service based on the file release prediction model; the service building module 40 builds a file release prediction service based on the file release prediction model. Users use the file release prediction service, and the user's operation is standardized and time-saving, which reduces the invalid data of its file release system. QA re-reviews the modification and release workload.

[0077] See also Figure 2 In one embodiment of the present invention, the preprocessing module 10 includes:

[0078] The first removal submodule 11 is used to remove dirty data in the release application history record according to the maintenance requirements of the file release system; specifically, the file release system maintenance personnel understand the requirements and the specific meaning of each field in the record, and remove the dirty data in the release application history record through the first removal submodule 11.

[0079] The gap filling submodule 12 is used to find the fields related to the vacant fields in the release application history record, and use the related fields to predict the content of the vacant fields, and give the vacant fields a default filling value. For the vacant fields in the record, the gap filling submodule 12 finds other fields related to the field, and uses other fields to predict the vacant fields, and gives the field a reasonable default filling value.

[0080] The feature extraction module 20 includes:

[0081] The second removal submodule 21 is used to remove fields irrelevant to the predicted file release from the historical records that meet the preset requirements; the second removal submodule 21 is used to select and remove some fields that are obviously useless for file release prediction.

[0082] The third removal submodule 22 is used to remove the units of the numerical fields in the historical records that meet the preset requirements, unify the dimensions of the numerical fields, and normalize the data to obtain the first type of numerical fields related to the release of the forecast file; the third removal submodule 22 removes the units of the numerical fields, unifies the dimensions, and normalizes the data. For example, if the current release ratio is 10%, remove %, and convert it into a float type value of 0.1.

[0083] The identification submodule 23 is used to use unique digital identification for the non-numeric fields in the historical records that meet the preset requirements to obtain the second type of numerical fields related to the release of the forecast file; the identification submodule 23 performs numerical operations on the non-numeric fields and identifies them with unique digital identification.

[0084] The evaluation submodule 24 is used to evaluate the contribution value of each field in the first category of numerical fields and the second category of numerical fields to the release of the predicted file using a preset second algorithm, and select valid fields that are effective for the release of the predicted file according to the contribution value, and mark the valid fields as record features. After the operation of the above modules, all the fields in the historical records that may be useful for the file release prediction are converted into numerical types.

[0085] The contribution value of the reserved fields of the evaluation submodule 24 to the file release prediction is further evaluated by means of the random forest algorithm, and the fields that are really useful for the file release prediction are screened out according to the contribution value, and these fields are the features of the record. After the above operation is performed on all historical records, a training data set is obtained. That is, the feature extraction module 20 completes the feature extraction of the historical records that meet the preset requirements, and all the valid fields marked as the record features obtained constitute the data set.

[0086] See also Figure 3 In one embodiment of the present invention, the model building module 30 includes:

[0087] A splitting submodule 31 is used to split the data set into a training set and a test set according to a preset ratio;

[0088] The data set obtained in step 2 is split into a training set and a test set in a ratio of 8:2.

[0089] The construction submodule 32 is used to construct the file release prediction model on the training set using a preset first algorithm; and to construct the file release prediction model on the training set using a GBDT algorithm.

[0090] The calculation submodule 33 is used to test the file release prediction model on the test set and calculate the prediction accuracy of the file release prediction model; test the obtained release prediction model on the test set, calculate the prediction accuracy of the model, and continuously adjust the model-related parameters according to the prediction effect.

[0091] The finalization submodule 34 is used to determine that when the accuracy of the file release prediction model reaches the expected effect, the file release prediction model is the final file release prediction model. When the model accuracy reaches the expected effect, the model is the final file release prediction model.

[0092] In addition, preferably, the model building module 30 also includes:

[0093] The parameter adjustment submodule 35 is used to continuously adjust the relevant parameters of the file release prediction model according to the expected effect after the prediction accuracy of the file release prediction model is calculated by the calculation submodule 33.

[0094] The preset first algorithm is a decision tree algorithm; the preset second algorithm is a random forest algorithm; the preset ratio is 8:2;

[0095] The prediction accuracy of the file release prediction model is calculated by the accuracy_score method in the sklearn library (sklearn is the abbreviation of scikit-learn, a third-party module based on Python); the file release prediction model related parameters include the learning rate, number of iterations and tree depth in the decision tree algorithm.

[0096] See also Figure 3 and Figure 4 In one embodiment of the present invention, the service building module 40 includes:

[0097] The first service building submodule 41 is used to build the file publishing prediction service using a preset framework and integrate it into the file publishing system in a preset connection method; for example, a file publishing prediction service is built using the tornado framework and integrated into the file publishing system in the form of a web interface.

[0098] The second service building submodule 42 is used for, when the file release prediction service receives a release prediction request, verifying whether the release prediction request has been predicted, and if so, directly extracting the prediction result of the file release prediction model from the cache of the file release system and returning it;

[0099] If not, the release prediction request is added to the task queue of the file release system. After getting the request from the task queue, the data in the release prediction request is parsed, and feature extraction is performed on the data for the historical records that meet the preset requirements. After the trainable data set is processed, it is input into the file release prediction model. After the file release prediction model gives a prediction value, it is returned and added to the cache.

[0100] When the file publishing prediction service receives a publishing prediction request, it first checks whether the request has been predicted. If it has been predicted, it directly gets the model prediction result from the cache and returns it. If it has not been predicted, it adds the request to the task queue. After the worker gets the request from the task queue, it parses the data in the request and performs Figure 4 After processing in step S402 shown, the data is input into the model, and the model returns and adds the predicted value to the cache.

[0101] Preferably, the service building module 40 further includes:

[0102] The optimization submodule 43 is used to periodically retrieve new file release records from a preset database, update the file release prediction model online, and optimize the file release prediction model.

[0103] In addition, the file release prediction service regularly retrieves new file release records from the database, updates the file release model online, and continuously optimizes the file release model. The preset framework is the Flask framework, the Django framework, and the Tornado framework; the preset connection method is the web interface method. The release prediction request is a prediction request for releasing a file.

[0104] File release prediction can be understood as a classification problem. The classic machine learning algorithms for solving classification problems include decision trees, support vector machines (SVM), and ensemble learning algorithms GBDT (GradientBoosting Decision Tree), also known as MART (Multiple Additive Regression Tree). It is an iterative decision tree algorithm that consists of multiple decision trees, and the conclusions of all trees are accumulated to make the final answer. When it was first proposed, it was considered to be an algorithm with strong generalization ability together with SVM. The tree in GBDT is a regression tree (not a classification tree). GBDT is used for regression prediction and can also be used for classification after adjustment. The idea of ​​GBDT gives it a natural advantage in discovering a variety of distinguishing features and feature combinations. In addition, the effect of the decision tree algorithm depends on the quality of the data. If the training data and file releases meet the same distribution, the decision tree will have a good classification effect. If the training data is too small or unevenly distributed, it is not enough to represent the entire file release problem. The constructed model is likely to be overfitted, that is, it works well on the training set and poorly on the test set. SVM can be applied to multiple classifications, but it is still a weak classifier, and the model constructed for unevenly distributed data sets is also not very effective. GBDT is an ensemble learning algorithm, which is a model composed of multiple decision trees. The algorithm will continuously adjust the proportion and weight of samples during the training process, which can solve the problem of uneven data distribution and obtain a robust classification model. Therefore, in one embodiment of the present invention, a file release prediction model is implemented using the GBDT algorithm to assist the 360 ​​file release system in realizing automatic file release.

[0105] In the above-mentioned multiple embodiments, the file release is predicted by using the system 100 for predicting file release, and the accuracy effect of the file release prediction model is: the file release prediction model needs to predict the file release object, ratio, and grayscale value, so a total of three prediction models are included, namely the file release object prediction model, the file release ratio prediction model, and the file release grayscale value prediction model. The accuracy of the three prediction models is 82%, 70%, and 98% respectively. The performance effect of the file release prediction service: after receiving the request, the file release prediction service needs to call the file release object prediction model, the file release ratio prediction model, and the file release grayscale value prediction model to predict the object, ratio, and grayscale value of the current application file. The file release prediction service can return the file release object, ratio, and grayscale value prediction value within 100ms.

[0106] See also Figure 4 In one embodiment of the present invention, a method for implementing a prediction file release using the prediction file release system 100 provided in any of the above embodiments is provided, including:

[0107] In step S401, the release application history records of the file release system are preprocessed to obtain the history records that meet the preset requirements;

[0108] In step S402, feature extraction is performed on the historical records that meet the preset requirements to obtain a trainable data set;

[0109] In step S403, a file release prediction model is constructed using the data set using a preset first algorithm;

[0110] In step S404, a file release prediction service is established according to the file release prediction model.

[0111] In one embodiment of the present invention, the preprocessing of the release application history records of the file release system includes:

[0112] According to the maintenance requirements of the document publishing system, dirty data in the historical records of the publishing application are removed to obtain historical data useful for document publishing;

[0113] For the vacant fields in the release application history record, find the fields related to the vacant fields, use the related fields to predict the content of the vacant fields, and give the vacant fields a default filling value.

[0114] The feature extraction of the historical records meeting the preset requirements includes:

[0115] Removing fields irrelevant to the release of the forecast file from the historical records that meet the preset requirements;

[0116] The units of the numerical fields in the historical records that meet the preset requirements are removed, and the dimensions of the numerical fields are unified and the data is normalized to obtain the first type of numerical fields related to the release of the prediction file; unifying the dimensions of the fields and normalizing the data will accelerate the convergence of the model.

[0117] Using unique digital identifiers for non-fields of the numerical type in the historical records that meet the preset requirements, to obtain fields of the second type of numerical type related to the release of the forecast file;

[0118] After the above three steps, all the fields in the historical records that may be useful for file release prediction are converted into numeric types. Since the machine learning algorithm used in the present invention can only receive fixed-length numeric matrix features, it is necessary to convert the character type into a numeric type in the data preprocessing stage.

[0119] A preset second algorithm is used to evaluate the contribution value of each field in the first category of numerical fields and the second category of numerical fields to the release of the prediction file, and valid fields that are effective for the release of the prediction file are selected based on the contribution values, and the valid fields are marked as record features.

[0120] In this step, feature selection is being performed. Feature selection can be roughly divided into two categories. One is a single feature selection method represented by the calculation of the Pearson coefficient, and the other is to score all features using machine learning methods. The available methods include regression models, SVM, decision trees, random forests, etc. Because random forests have good robustness, this embodiment is the first choice for feature selection using machine learning algorithms. For the determination of contribution values, since this calculation book solution belongs to a classification problem, the RandomForestClassifier method in the sklearn library is used to determine the impact of all features on classification.

[0121] Example of determining contribution value:

[0122] forest=RandomForestClassifier(n_estimators=200,random_state=0)

[0123] forest.fit(X_train,Y_train)

[0124] importances=forest.feature_importances_

[0125] Among them, X_train represents the samples in the training set. A sample consists of multiple features. Y_train represents the category to which each sample belongs. Importances is the contribution value of each feature in the sample to the classification calculated by random forest. The larger the value, the more important the feature is and the greater the impact on the classification effect.

[0126] The obtaining of the trainable data set includes: completing feature extraction of the historical records that meet the preset requirements, and all the valid fields marked as the record features obtained constitute the data set.

[0127] Specifically, the step of using the preset first algorithm to construct a file release prediction model using the data set includes:

[0128] The data set is split into a training set and a test set according to a preset ratio; usually, a part of the data set is separated as a training set for model training, and a part is separated as a test set for model testing. The training set is larger than the test set, and the ratio can be 6:4, 7:3, 8:2, 9:1, etc. In this embodiment, 8:2 is selected.

[0129] Constructing the file release prediction model using a preset first algorithm on the training set;

[0130] Testing the file release prediction model on the test set, and calculating the prediction accuracy of the file release prediction model;

[0131] When the accuracy of the file release prediction model reaches the expected effect, the file release prediction model is the final file release prediction model. In this embodiment, the accuracy_score method in the sklearn library is used. Because this technical solution belongs to a classification problem, the prediction accuracy refers to the proportion of the correct prediction of the category in the test set to the entire test set.

[0132] After calculating the prediction accuracy of the file release prediction model, the method further includes: continuously adjusting the parameters related to the file release prediction model according to the expected effect. In this embodiment, the model training of this technical solution uses the GBDT algorithm encapsulated by sklearn, and the adjustable parameters include learning rate, number of iterations, tree depth, etc. These parameters are all hyperparameters, and they need to be continuously adjusted through experiments to determine the values ​​of these parameters according to the effect of the model.

[0133] Preferably in the present invention, the preset first algorithm is a decision tree algorithm; the preset second algorithm is a random forest algorithm; the preset ratio is 8:2; the prediction accuracy of the file release prediction model is calculated by the accuracy_score method in the sklearn library; the file release prediction model related parameters include the learning rate, number of iterations and tree depth in the decision tree algorithm.

[0134] In one embodiment of the present invention, the building of a file release prediction service according to the file release prediction model includes:

[0135] The file publishing prediction service is built using a preset framework and integrated into the file publishing system in a preset connection method; this technical solution is implemented and developed based on Python, and the web frameworks that can be used in Python include Flask, Django, Tornado, etc.

[0136] When the file publishing prediction service receives a publishing prediction request, it verifies whether the publishing prediction request has been predicted, and if so, directly extracts the prediction result of the file publishing prediction model from the cache of the file publishing system and returns it;

[0137] If not, the release prediction request is added to the task queue of the file release system. After getting the request from the task queue, the data in the release prediction request is parsed, and feature extraction is performed on the data for the historical records that meet the preset requirements. After the trainable data set is processed, it is input into the file release prediction model. After the file release prediction model gives a prediction value, it is returned and added to the cache.

[0138] Finally, the file release prediction service built according to the file release prediction model further includes:

[0139] New file release records are retrieved regularly from a preset database, the file release prediction model is updated online, and the file release prediction model is optimized. The preset framework is a Flask framework, a Django framework, and a Tornado framework; and the preset connection method is a web interface method.

[0140] join Figure 5 and Figure 6In one embodiment of the present invention, a storage medium 601 is provided for storing a computer program for executing any one of the methods for predicting file release in any one of the above embodiments. A computer device 500 is also provided, comprising a memory 502, a processor 501, and a computer program stored in the memory 502 and executable on the processor 501, wherein the processor implements any one of the methods for predicting file release described above when executing the computer program.

[0141] The present invention also provides a storage medium for storing Figure 4 A computer program of any of the methods described. For example, a computer program instruction, when executed by a computer, can call or provide the method and / or technical solution according to the present application through the operation of the computer. The program instruction for calling the method of the present application may be stored in a fixed or removable storage medium, and / or transmitted through a data stream in a broadcast or other signal-bearing medium and / or stored in a storage medium 601 of a computer device running according to the program instruction. Here, according to an embodiment of the present application, a computer program includes a computer program such as Figure 5 The computer device 500 shown in FIG. 1 preferably includes a memory 502 for storing a computer program and a processor 501 for executing the computer program. The memory 502 has a storage space 503, such as Figure 6 As shown, the memory 502 is a computer-readable storage medium 601, which stores program codes for executing the method steps of the present invention. When the computer program is executed by the processor 501, the computer device 500 is triggered to execute the methods and / or technical solutions based on the aforementioned multiple embodiments.

[0142] It should be noted that the present application can be implemented in software and / or a combination of software and hardware, for example, can be implemented using an application specific integrated circuit (ASIC), a general purpose computer or any other similar hardware device. In one embodiment, the software program of the present application can be executed by a processor to implement the above steps or functions. Similarly, the software program of the present application (including relevant data structures) can be stored in a computer-readable recording medium, for example, a RAM memory, a magnetic or optical drive or a floppy disk and similar devices. In addition, some steps or functions of the present application can be implemented using hardware, for example, as a circuit that cooperates with a processor to perform each step or function.

[0143] The method according to the present invention can be implemented on a computer as a computer-implemented method, or implemented in dedicated hardware, or implemented in a combination of the two. The executable code or part thereof for the method according to the present invention can be stored on a computer program product. The example of a computer program product includes a memory device, an optical storage device, an integrated circuit, a server, online software, etc. Preferably, the computer program product includes a non-temporary program code component stored on a computer-readable medium so as to perform the method according to the present invention when the program product is executed on a computer.

[0144] In a preferred embodiment, the computer program comprises computer program code means adapted to perform all the steps of the method according to the invention when the computer program is run on a computer.Preferably, the computer program is embodied on a computer readable medium.

[0145] In summary, the present invention first preprocesses the release application history records of the file release system, performs feature extraction on the history records that meet the preset requirements, and after obtaining a trainable data set, uses the data set to construct a file release prediction model using a preset first algorithm; finally, a file release prediction service is built based on the file release prediction model, thereby optimizing the release management operations of file release and updates, and reducing the workload of managers for file release. Through the provision of the present invention, managers can provide specifications for the objects and quantities of newly launched or updated files during the submission of release applications, remove invalid data, and do not need QA to review and modify again, thereby reducing the workload of release. The present invention implements an automated file release solution through the GBDT algorithm, which solves the pain points of file release systems in the prior art.

[0146] Of course, the present invention may have many other embodiments. Without departing from the spirit and essence of the present invention, those skilled in the art may make various corresponding changes and modifications based on the present invention, but these corresponding changes and modifications should all fall within the scope of protection of the claims attached to the present invention.

[0147] The present invention discloses A1, a method for predicting file release, comprising:

[0148] Pre-process the release application history records of the document release system to obtain the history records that meet the preset requirements;

[0149] Extracting features from the historical records that meet the preset requirements to obtain a trainable data set;

[0150] Using a preset first algorithm and the data set to construct a file release prediction model;

[0151] A file release prediction service is established based on the file release prediction model.

[0152] A2. According to the method described in A1, the preprocessing of the release application history records of the file release system includes:

[0153] According to the maintenance requirements of the file publishing system, remove the dirty data in the publishing application history record;

[0154] For the vacant fields in the release application history record, find fields related to the vacant fields, use the related fields to predict the content of the vacant fields, and give the vacant fields a default filling value;

[0155] The feature extraction of the historical records meeting the preset requirements includes:

[0156] Removing fields irrelevant to the release of the forecast file from the historical records that meet the preset requirements;

[0157] The units of the numerical fields in the historical records that meet the preset requirements are removed, and the dimensions of the numerical fields are unified and the data is normalized to obtain the first type of numerical fields related to the release of the forecast file;

[0158] Using unique digital identifiers for non-fields of the numerical type in the historical records that meet the preset requirements, to obtain fields of the second type of numerical type related to the release of the forecast file;

[0159] Using a preset second algorithm to evaluate the contribution value of each field in the first category of numerical fields and the second category of numerical fields to the release of the prediction file, and selecting valid fields that are effective for the release of the prediction file according to the contribution values, and marking the valid fields as record features;

[0160] The step of obtaining a trainable data set comprises:

[0161] Complete feature extraction of the historical records that meet the preset requirements, and obtain all valid fields marked as the record features to form the data set;

[0162] The step of using the preset first algorithm to construct a file release prediction model using the data set includes:

[0163] Splitting the data set into a training set and a test set according to a preset ratio;

[0164] Constructing the file release prediction model using a preset first algorithm on the training set;

[0165] Testing the file release prediction model on the test set, and calculating the prediction accuracy of the file release prediction model;

[0166] When the accuracy of the file release prediction model reaches the expected effect, the file release prediction model is the final file release prediction model.

[0167] A3. The method according to A2, after calculating the prediction accuracy of the file release prediction model, further comprising: continuously adjusting the relevant parameters of the file release prediction model according to the expected effect;

[0168] The preset first algorithm is a decision tree algorithm; the preset second algorithm is a random forest algorithm; the preset ratio is 8:2;

[0169] The prediction accuracy of the file release prediction model is calculated using the accuracy_score method in the sklearn library;

[0170] The file release prediction model related parameters include the learning rate, the number of iterations and the depth of the tree in the decision tree algorithm.

[0171] A4. According to the method described in A1, building a file release prediction service based on the file release prediction model includes:

[0172] Using a preset framework to build the document publishing prediction service, and integrating it into the document publishing system in a preset connection mode;

[0173] When the file publishing prediction service receives a publishing prediction request, it verifies whether the publishing prediction request has been predicted, and if so, directly extracts the prediction result of the file publishing prediction model from the cache of the file publishing system and returns it;

[0174] If not, the release prediction request is added to the task queue of the file release system. After getting the request from the task queue, the data in the release prediction request is parsed, and feature extraction is performed on the data for the historical records that meet the preset requirements. After the trainable data set is processed, it is input into the file release prediction model. After the file release prediction model gives a prediction value, it is returned and added to the cache.

[0175] A5. The method according to A4, wherein the building of a file release prediction service according to the file release prediction model further comprises:

[0176] Regularly extract new file release records from a preset database, update the file release prediction model online, and optimize the file release prediction model;

[0177] The types of file release prediction models include: file release object prediction model, file release ratio prediction model, and file release gray value prediction model;

[0178] The preset frameworks are Flask framework, Django framework and Tornado framework; the preset connection method is the web interface method.

[0179] The present invention also discloses B6, a system for predicting file release, comprising:

[0180] A preprocessing module is used to preprocess the release application history records of the file release system to obtain the history records that meet the preset requirements;

[0181] A feature extraction module is used to extract features from the historical records that meet the preset requirements to obtain a trainable data set;

[0182] A model building module, used to use a preset first algorithm to build a file release prediction model using the data set;

[0183] The service building module is used to build a file release prediction service based on the file release prediction model.

[0184] B7. According to the system described in B6, the preprocessing module comprises:

[0185] A first removal submodule is used to remove dirty data in the release application history records according to the maintenance requirements of the file release system;

[0186] A gap filling submodule is used to find the fields related to the vacant fields in the release application history record, predict the content of the vacant fields by using the related fields, and fill the vacant fields with default values;

[0187] The feature extraction module comprises:

[0188] A second removal submodule, configured to remove fields irrelevant to the release of the prediction file from the historical records that meet the preset requirements;

[0189] A third removal submodule is used to remove units from the numerical fields in the historical records that meet the preset requirements, and to unify the dimensions and perform data normalization on the numerical fields to obtain the first type of numerical fields related to the release of the forecast file;

[0190] An identification submodule, used for using a unique digital identification for the non-fields of the numerical type in the historical records that meet the preset requirements, to obtain the fields of the second type of numerical type related to the release of the forecast file;

[0191] An evaluation submodule, configured to evaluate the contribution value of each field in the first category of numerical fields and the second category of numerical fields to the release of the prediction file by using a preset second algorithm, and select valid fields that are effective for the release of the prediction file according to the contribution values, and mark the valid fields as record features;

[0192] The feature extraction module completes feature extraction of the historical records that meet the preset requirements, and all valid fields marked as the record features obtained constitute the data set;

[0193] The model building module includes:

[0194] A splitting submodule, used to split the data set into a training set and a test set according to a preset ratio;

[0195] A construction submodule, configured to construct the file release prediction model using a preset first algorithm on the training set;

[0196] A calculation submodule, used for testing the file release prediction model on the test set and calculating the prediction accuracy of the file release prediction model;

[0197] The finalization submodule is used to determine that when the accuracy of the file release prediction model reaches the expected effect, the file release prediction model is the final file release prediction model.

[0198] B8. According to the system described in B7, the model building module also includes:

[0199] A parameter adjustment submodule, configured to continuously adjust relevant parameters of the file release prediction model according to the expected effect after the prediction accuracy of the file release prediction model is calculated by the calculation submodule;

[0200] The preset first algorithm is a decision tree algorithm; the preset second algorithm is a random forest algorithm; the preset ratio is 8:2;

[0201] The prediction accuracy of the file release prediction model is calculated using the accuracy_score method in the sklearn library;

[0202] The file release prediction model related parameters include the learning rate, the number of iterations and the depth of the tree in the decision tree algorithm.

[0203] B9. According to the system described in B8, the service building module includes:

[0204] A construction submodule is used to construct the file publishing prediction service using a preset framework and integrate it into the file publishing system in a preset connection mode;

[0205] A service building submodule is used for, when the file publishing prediction service receives a publishing prediction request, verifying whether the publishing prediction request has been predicted, and if so, directly extracting the prediction result of the file publishing prediction model from the cache of the file publishing system and returning it;

[0206] If not, the release prediction request is added to the task queue of the file release system. After getting the request from the task queue, the data in the release prediction request is parsed, and feature extraction is performed on the data for the historical records that meet the preset requirements. After the trainable data set is processed, it is input into the file release prediction model. After the file release prediction model gives a prediction value, it is returned and added to the cache.

[0207] B10. According to the system described in B9, the service building module further includes:

[0208] An optimization submodule, used for periodically retrieving new file release records from a preset database, updating the file release prediction model online, and optimizing the file release prediction model;

[0209] The types of the file release prediction model include: a file release object prediction model, a file release ratio prediction model, and a file release gray value prediction model.

[0210] B11 According to the system described in B10, the preset framework is Flask framework, Django framework and Tornado framework; the preset connection method is a web interface method.

[0211] The present invention discloses C12, a storage medium for storing a computer program for executing any one of the methods for predicting file release in A1-A5.

[0212] The present invention discloses C13, a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method for predicting file release described in any one of A1 to A5 when executing the computer program.

Claims

1. A method for predicting document release, It is characterized in that include: Pre-process the release application history records of the document release system to obtain the history records that meet the preset requirements; Extracting features from the historical records that meet the preset requirements to obtain a trainable data set; Using a preset first algorithm and the data set to construct a file release prediction model; A file release prediction service is established based on the file release prediction model.

2. The method according to claim 1, It is characterized in that The preprocessing of the release application history record of the file release system includes: According to the maintenance requirements of the file publishing system, remove the dirty data in the publishing application history record; For the vacant fields in the release application history record, find fields related to the vacant fields, use the related fields to predict the content of the vacant fields, and give the vacant fields a default filling value; The feature extraction of the historical records meeting the preset requirements includes: Removing fields irrelevant to the release of the forecast file from the historical records that meet the preset requirements; The units of the numerical fields in the historical records that meet the preset requirements are removed, and the dimensions of the numerical fields are unified and the data is normalized to obtain the first type of numerical fields related to the release of the forecast file; Using unique digital identifiers for non-fields of the numerical type in the historical records that meet the preset requirements, to obtain fields of the second type of numerical type related to the release of the forecast file; Using a preset second algorithm to evaluate the contribution value of each field in the first category of numerical fields and the second category of numerical fields to the release of the prediction file, and selecting valid fields that are effective for the release of the prediction file according to the contribution values, and marking the valid fields as record features; The step of obtaining a trainable data set comprises: Complete feature extraction of the historical records that meet the preset requirements, and obtain all valid fields marked as the record features to form the data set; The step of using the preset first algorithm to construct a file release prediction model using the data set includes: Splitting the data set into a training set and a test set according to a preset ratio; Constructing the file release prediction model using a preset first algorithm on the training set; Testing the file release prediction model on the test set, and calculating the prediction accuracy of the file release prediction model; When the accuracy of the file release prediction model reaches the expected effect, the file release prediction model is the final file release prediction model.

3. The method according to claim 2, It is characterized in that After calculating the prediction accuracy of the file release prediction model, the method further includes: continuously adjusting the relevant parameters of the file release prediction model according to the expected effect; the preset first algorithm is a decision tree algorithm; the preset second algorithm is a random forest algorithm; the preset ratio is 8:2; The prediction accuracy of the file release prediction model is calculated using the accuracy_score method in the sklearn library; The file release prediction model related parameters include the learning rate, the number of iterations and the depth of the tree in the decision tree algorithm.

4. The method according to claim 1, It is characterized in that The building of a file release prediction service according to the file release prediction model includes: Using a preset framework to build the document publishing prediction service, and integrating it into the document publishing system in a preset connection mode; When the file publishing prediction service receives a publishing prediction request, it verifies whether the publishing prediction request has been predicted, and if so, directly extracts the prediction result of the file publishing prediction model from the cache of the file publishing system and returns it; If not, the release prediction request is added to the task queue of the file release system. After getting the request from the task queue, the data in the release prediction request is parsed, and feature extraction is performed on the data for the historical records that meet the preset requirements. After the trainable data set is processed, it is input into the file release prediction model. After the file release prediction model gives a prediction value, it is returned and added to the cache.

5. The method according to claim 4, It is characterized in that The building of the file release prediction service according to the file release prediction model further includes: Regularly extract new file release records from a preset database, update the file release prediction model online, and optimize the file release prediction model; The types of file release prediction models include: file release object prediction model, file release ratio prediction model, file release gray value prediction model; the preset frameworks are Flask framework, Django framework and Tornado framework; the preset connection method is a web interface method.

6. A system for predicting document release, It is characterized in that include: A preprocessing module is used to preprocess the release application history records of the file release system to obtain the history records that meet the preset requirements; A feature extraction module is used to extract features from the historical records that meet the preset requirements to obtain a trainable data set; A model building module, used to use a preset first algorithm to build a file release prediction model using the data set; The service building module is used to build a file release prediction service based on the file release prediction model.

7. The system according to claim 6, It is characterized in that The preprocessing module comprises: A first removal submodule, used to remove dirty data in the release application history records according to the maintenance requirements of the file release system; A gap filling submodule is used to find the fields related to the vacant fields in the release application history record, predict the content of the vacant fields by using the related fields, and fill the vacant fields with default values; The feature extraction module comprises: A second removal submodule, configured to remove fields irrelevant to the release of the prediction file from the historical records that meet the preset requirements; A third removal submodule is used to remove units from the numerical fields in the historical records that meet the preset requirements, and to unify the dimensions and perform data normalization on the numerical fields to obtain the first type of numerical fields related to the release of the forecast file; An identification submodule, used for using a unique digital identification for the non-fields of the numerical type in the historical records that meet the preset requirements, to obtain the fields of the second type of numerical type related to the release of the forecast file; An evaluation submodule, configured to evaluate the contribution value of each field in the first category of numerical fields and the second category of numerical fields to the release of the prediction file by using a preset second algorithm, and select valid fields that are effective for the release of the prediction file according to the contribution values, and mark the valid fields as record features; The feature extraction module completes feature extraction of the historical records that meet the preset requirements, and all valid fields marked as the record features obtained constitute the data set; The model building module includes: A splitting submodule, used to split the data set into a training set and a test set according to a preset ratio; A construction submodule, configured to construct the file release prediction model using a preset first algorithm on the training set; A calculation submodule, used for testing the file release prediction model on the test set and calculating the prediction accuracy of the file release prediction model; The finalization submodule is used to determine that when the accuracy of the file release prediction model reaches the expected effect, the file release prediction model is the final file release prediction model.

8. The system according to claim 7, It is characterized in that The service building module includes: A construction submodule is used to construct the file publishing prediction service using a preset framework and integrate it into the file publishing system in a preset connection mode; A service building submodule is used for, when the file publishing prediction service receives a publishing prediction request, verifying whether the publishing prediction request has been predicted, and if so, directly extracting the prediction result of the file publishing prediction model from the cache of the file publishing system and returning it; If not, the release prediction request is added to the task queue of the file release system. After getting the request from the task queue, the data in the release prediction request is parsed, and feature extraction is performed on the data for the historical records that meet the preset requirements. After the trainable data set is processed, it is input into the file release prediction model. After the file release prediction model gives a prediction value, it is returned and added to the cache.

9. A storage medium, It is characterized in that A computer program for storing a computer program for executing the method for predicting document release in any one of claims 1 to 5.

10. A computer device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, It is characterized in that When the processor executes the computer program, the method for predicting file release according to any one of claims 1 to 5 is implemented.