Prediction model optimization method and system for DSP platform

By creating an experimental directory and data synchronization protocol on the DSP platform, data acquisition and preprocessing are automated, the data acquisition and input and output processes are optimized, the data fragmentation problem is solved, the accuracy and efficiency of the prediction model are improved, and multi-scenario adaptability is supported.

CN120654429AActive Publication Date: 2025-09-16GUANGZHOU TAIDONG TECH CO LTD
View PDF 1 Cites 0 Cited by

Patent Information

Application Number
CN202510834740.3
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2025-06-20
Publication Date
2025-09-16
Estimated Expiration
2045-06-20

AI Technical Summary

Technical Problem

In the DSP platform, existing technologies have the problem of fragmented data acquisition, preprocessing and model input and output, resulting in low prediction efficiency and accuracy of the prediction model. Especially in the scenario of multi-source heterogeneous data, manual operations can easily lead to data omissions or inconsistent formats, affecting the accuracy of model training.

Method used

By creating an experimental directory, automatically pulling sample data based on the data synchronization protocol, performing preprocessing and standardization, optimizing data acquisition and input and output processes, and providing a universal interface to support seamless access to different machine learning models, automated data management and model flexibility can be achieved.

Benefits of technology

It improves the accuracy and efficiency of the prediction model, avoids data pollution and file conflicts, supports the expansion of prediction scenarios in multiple industries, and achieves the reproducibility and adaptability of model training.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN120654429A_ABST
    Figure CN120654429A_ABST
Patent Text Reader

Abstract

The invention relates to the technical field of data processing, in particular to an optimization method and system for a prediction model of a DSP platform. The method comprises the following steps: creating an experiment directory; based on a data synchronization protocol, sample data are pulled from a preset pre-stored data directory or a remote database to the experiment directory; preprocessing a target file under the experiment directory to obtain preprocessed data; performing standardization based on a preset rule for unifying an input format, then obtaining the preprocessed data based on a standardization result, inputting the preprocessed data into a prediction model for training and testing, and outputting a prediction result according to a set path; the prediction result comprises a prediction value and a true value, and the prediction model is a machine learning model. According to the method, the prediction accuracy of the prediction model can be improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and more particularly to an optimization method and system for a prediction model used in a DSP platform. Background Art

[0002] In scenarios such as demand-side platforms (DSPs), obtaining the data required for predictive models often relies on manual operations, especially when it comes to multi-source heterogeneous data (such as transaction dates, transaction events, commission amounts, etc.). These data are usually stored in databases or file systems in a scattered manner and need to be manually downloaded and preprocessed (need to manually write corresponding processing rules, and then process the data to be processed one by one according to the processing rules). This is not only time-consuming, but also prone to data omissions or inconsistent formats due to non-standard operations, which in turn affects the accuracy of model training.

[0003] Therefore, how to optimize the process of obtaining input data to avoid data omissions or inconsistent formats due to human errors in order to improve the accuracy of model predictions is a technical problem that needs to be solved urgently. Summary of the Invention

[0004] In order to solve the technical problem of low prediction accuracy of the model caused by human error, the present invention provides solutions in the following aspects.

[0005] In the first aspect, the present invention provides an optimization method for a prediction model for a DSP platform, comprising: creating an experimental directory; pulling sample data from a preset pre-stored data directory or a remote database to the experimental directory based on a data synchronization protocol; preprocessing the target files under the experimental directory to obtain preprocessed data; standardizing based on preset rules for unifying the input format, and then obtaining the preprocessed data based on the standardized results, and inputting it into the prediction model for training and testing, and outputting the prediction results according to the set path; the prediction results include predicted values ​​and true values, and the prediction model is a machine learning model.

[0006] Beneficial effects: By standardizing and automating data acquisition, preprocessing, and input and output processes, the process of obtaining data required for model training and testing is optimized, avoiding problems such as inconsistent data formats and data pollution, thereby improving the accuracy and efficiency of predictive model predictions. It also provides a universal interface for model training, enabling simple and rapid model replacement.

[0007] Furthermore, creating an experiment directory includes: creating the experiment directory using an experiment root directory path and an experiment identifier.

[0008] Beneficial effects: By creating an experimental directory and obtaining data based on the experimental directory, data contamination can be avoided and the reproducibility of the experiment can be ensured.

[0009] Furthermore, pulling sample data from a preset pre-stored data directory or remote database to the experimental directory includes: obtaining a date range, the date range including a training set date range and a test set date range; for any target date in the date range, determining whether there is a sample data file corresponding to the target date in the pre-stored data directory, and if so, copying the sample data file from the pre-stored data directory to the training set file directory or the test set file directory under the experimental directory; if not, pulling sample data from the remote database according to the first granularity of the target date to obtain a first-granularity sample data file; aggregating all first-granularity sample data files corresponding to the target date into a second-granularity sample data file, and storing it in the training set file directory or the test set file directory under the experimental directory.

[0010] Beneficial effects: By automatically acquiring and standardizing sample data, data contamination caused by manual operation errors is avoided, and the efficiency and accuracy of data acquisition are optimized.

[0011] Furthermore, the target files in the experimental directory are preprocessed, including: for the files in the training set file directory, all files are spliced ​​to obtain the training set files; for the files in the test set file directory, all files are spliced ​​to obtain the test set files; the first target column in the training set file and the test set file is deleted, and the second target column with missing values ​​is filled.

[0012] Furthermore, the standardized result includes a training set file path, a test set file path, a first target column and a second target column; obtaining the preprocessed data based on the standardized result includes: obtaining test data according to the test set file path; obtaining training data according to the training set file path; obtaining a column to be deleted according to the first target column, and obtaining a column to be predicted according to the second target column.

[0013] Furthermore, before creating the experiment directory, the process also includes: clearing the visualization result directory and the original data directory.

[0014] Furthermore, the prediction result also includes a conclusion file for recording model evaluation parameters, and a model metadata file for recording model type and hyperparameters; the optimization method also includes: selecting the optimal prediction model from the candidate prediction models based on the model evaluation parameters recorded in the conclusion file.

[0015] Furthermore, outputting the prediction results according to a set path includes: determining the set path according to input parameters of the input prediction model, and the input parameters include a prediction result output directory.

[0016] Furthermore, it also includes: after sorting the test results of the test set according to the true values, performing stratified sampling according to a preset stratified sampling strategy, and then drawing and outputting a comparison chart between the predicted values ​​and the true values.

[0017] Beneficial effects: By displaying performance on different data segments through stratified sampling, relevant personnel can quickly and accurately evaluate the model's prediction accuracy and potential bias, allowing timely adjustments to ensure the flexibility and adaptability of the prediction task.

[0018] In a second aspect, the present invention provides an optimization system for a prediction model for a DSP platform, comprising a processor and a memory, wherein the memory stores computer program instructions, and when the computer program instructions are executed by the processor, the optimization method for the prediction model for the DSP platform described in the first aspect is implemented.

[0019] The beneficial effects of the present invention are as follows: the present invention optimizes the acquisition / download of historical data by creating an experimental directory, avoiding the problem of data omission or inconsistent format caused by manual operation; by standardizing the data preprocessing process, the efficiency and accuracy of data preprocessing are optimized compared to manual preprocessing; by standardizing the model input and output processes, when replacing a model, only the definition of the model needs to be changed, without changing the input and output parts, which can meet the seamless access of different machine learning models and optimize the flexibility and adaptability of selecting prediction model types. By optimizing the data download, preprocessing and input and output processes, it is only necessary to adjust the data acquisition path and target column definition to expand to multiple industry prediction scenarios, thereby optimizing the scalability of model prediction. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 is a flow chart schematically illustrating a method for optimizing a prediction model for a DSP platform according to an embodiment of the present invention;

[0021] Figure 2 FIG. 4 is a block diagram schematically illustrating a structure of an optimization system for a prediction model of a DSP platform according to an embodiment of the present invention. DETAILED DESCRIPTION

[0022] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work shall fall within the scope of protection of the present invention.

[0023] The specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0024] Figure 1 FIG. 4 is a flow chart schematically illustrating a method for optimizing a prediction model for a DSP platform according to an embodiment of the present invention.

[0025] The current DSP platform's prediction model training has the following main defects:

[0026] 1. Manual data export, cleaning, and feature engineering. Model training relies on independent scripts, and data flow is achieved through manual file transfer. However, this method, which requires manual intervention in data export, cleaning, and parameter configuration, is time-consuming and prone to data contamination due to operational errors.

[0027] 2. Use ETL (Extract-Transform-Load) tools like Informatica (a core platform for data integration and management) to preprocess data, output it to platforms like TensorFlow / PyTorch for modeling, and visualize the results using BI (Business Intelligence) tools like Tableau. However, the disconnect between ETL tools and modeling platforms leads to a high failure rate in experimental replication (due to missing data tracebacks).

[0028] 3. Hyperparameter optimization and experiment tracking are implemented based on frameworks such as MLflow (a full lifecycle management framework for machine learning), supporting basic indicator output. However, the AutoML (Automated Machine Learning) system does not integrate a date-driven mechanism and cannot automatically respond to dynamic data updates (for example, new advertising channel data requires manual reconfiguration and collection).

[0029] 4. The DSP platform has a built-in SQL (Structured Query Language) query module that calculates historical commissions and uses simple linear prediction. However, the DSP platform uses simple models like linear regression, which results in high error rates and low prediction accuracy when modeling complex factors like ad type and time period weighting.

[0030] In addition, the existing solution lacks isolation of the experimental environment, and file conflicts are prone to occur when multiple tasks are performed in parallel.

[0031] In summary, the existing technology has problems with data acquisition, preprocessing, and fragmentation and dispersion of model input and output when training prediction models, which leads to low prediction efficiency and accuracy of the prediction models.

[0032] In this regard, in the first aspect, the present invention provides an optimization method for a prediction model for a DSP platform. This method realizes intelligent management of the entire life cycle from data update, model training to result analysis by building a deep coupling mechanism of data-model-analysis, thereby effectively improving prediction accuracy and efficiency. Moreover, by optimizing the data acquisition, preprocessing, and input and output processes, it is only necessary to adjust the acquisition path and target column definition to be applicable to various prediction scenarios involving multi-source heterogeneous data, such as prediction scenarios for predicting order commissions, prediction scenarios for predicting sales, prediction scenarios for predicting risks, prediction scenarios for predicting order volumes, etc. The specific implementation of the present invention will be described below using the scenario of order commission prediction as an example.

[0033] Specifically, if Figure 1 As shown, the method of the present invention includes:

[0034] S101. Create an experiment directory; based on a data synchronization protocol, pull sample data from a preset data directory or database to the experiment directory.

[0035] In one embodiment, to ensure the reproducibility of the experiment, the data storage path and experiment identifier need to be strictly standardized. Specifically, the experimental parameter space and date range are defined.

[0036] In this embodiment, the experiment parameter space includes the experiment root directory path, the experiment identifier (e.g., exp_0), and database connection parameters. The experiment root directory path is used to centrally manage all experiment input and output files; the experiment identifier is used to distinguish different experiment versions or configurations; and the database connection parameters are used to dynamically obtain non-localized data (i.e., pull sample data from a remote database).

[0037] Specifically, the experimental parameter space Ω is defined as:

[0038] Ω={base_path,exp_id,db_host,db_port};

[0039] Among them, base_path is the experiment root directory path, exp_id is the experiment identifier, db_host is the database host, and db_port is the database port.

[0040] In this embodiment, the date range includes the training set date range and the test set date range. Specifically, the date range D is defined as:

[0041] D=[train_start_date,train_end_date]∪{test_date};

[0042] Where train_start_date is the start date of the training set date range, train_end_date is the end date of the training set date range, and test_date is the date of the test set date range. It should be noted that in this embodiment, the test set date range can be a single date, or it can be composed of a start date and an end date like the training set date range.

[0043] Furthermore, an experiment directory is created according to the experiment root directory path and the experiment identifier in the experiment parameter space.

[0044] By defining the experimental parameter space and date range and building a complete experimental directory structure, data contamination between multiple experiments can be avoided, thereby ensuring the accuracy of the prediction model.

[0045] In one embodiment, the method of the present invention further includes clearing historical data. Specifically, the visualization results directory (commission_fig_path) and the original data directory (data_path) are cleared. By clearing historical data, residual data can be prevented from interfering with new experiments, resolving file conflicts that can easily arise when running multiple tasks in parallel and optimizing the accuracy of model predictions.

[0046] Furthermore, based on the data synchronization protocol, data is automatically pulled and aggregated from a preset pre-stored data directory or remote database.

[0047] Specifically, for any target date in the date range, determine whether there is a sample data file corresponding to the target date in the preset pre-stored data directory (i.e., local cache). If so, directly call the copy_files_by_date function to copy the sample data file corresponding to the target date in the pre-stored data directory to the training set file directory or the test set file directory under the experimental directory (if the target date falls within the training set date range, copy it to the training set file directory; if the target date falls within the test set date range, copy it to the test set file directory); if it does not exist, pull data according to the first granularity of the target date (in this embodiment, the first granularity is the hourly granularity) to obtain a first-granularity sample data file, and store it in a temporary directory (automatically created); aggregate all first-granularity sample data files corresponding to the target date in the temporary directory into a second-granularity sample data file (in this embodiment, the second granularity is the day granularity), and store it in the training set file directory or the test set file directory under the experimental directory.

[0048] Specifically, the pulling process is:

[0049]

[0050] Where d is the target date, D is the date range, source_dir is the pre-stored data directory, d.csv is the sample data file corresponding to the target date (for example, 2025-01-01.csv), copy_files_by_date() is a copy function that copies files by date, execute_sql() is a function that pulls data from a remote database, and temp_dir is a temporary directory used to store raw data files generated hourly (i.e., hourly granularity sample data files); d h .csv is the hourly granularity sample data file obtained at the hth hour (for example, 2025-01-01_00.csv), and → indicates generation and output.

[0051] Specifically, the polymerization process is:

[0052]

[0053] Where agg() is an aggregation function, d is the target date, d∈D, temp_dir is the temporary directory, train_target_dir is the training set file directory (if the target date falls within the test set date range, the corresponding file is stored in the test set file directory), d h .csv is the hourly granularity data file pulled at the hth hour, and d.csv is the daily granularity data file obtained by aggregation (that is, the sample data file corresponding to the target date).

[0054] It is understandable that the traditional method is to manually download and export these hourly granularity data files from a remote database and then aggregate them one by one. However, this method is time-consuming and labor-intensive, and manual operation is prone to errors, resulting in problems such as data contamination. In this regard, the method of the present application can achieve standardized automatic downloading (pulling) of data, avoiding data contamination caused by manual errors and improving the efficiency and accuracy of data processing.

[0055] Through the above process, the data set (including test set and training set) for model training and testing was obtained.

[0056] S102: Preprocess the target files in the experimental directory to obtain preprocessed data.

[0057] Since the original data may have inconsistent formats or redundant fields (columns), the original data can be preprocessed to generate a standardized data set that meets the model input specifications.

[0058] Specifically, for the files in the training set file directory, all files are concatenated (i.e., merged) row by row to obtain a complete training set file (train.csv) and save it in the data directory. Specifically, the merging process is:

[0059]

[0060] Among them, merge_csv_files() is the merge function, train_target_dir is the training set file directory, data_path is the data directory, and train.csv is the training set file.

[0061] The above merging process means that after executing merge_csv_files(train_target_dir), a file named train.csv is generated (output) under data_path.

[0062] Similarly, for the files in the test set file directory, all files are spliced ​​line by line to obtain a complete test set file (test.csv), and stored in the data directory. The specific merging process is the same or similar to that of the training set file and will not be repeated here.

[0063] Furthermore, the first target column (representing fields irrelevant to prediction, such as log ID, debug flag, etc.) in the training set file and / or the test set file is filtered out. By removing the first target column, data leakage or interference can be avoided.

[0064] Furthermore, missing values ​​are filled. Specifically, if the second target column (representing the variable or field to be predicted, such as commission) is missing in the training set file and / or the test set file, the second target column is filled. In this example, the second target column includes commission. If commission is missing, the field containing the missing commission is filled with the mean value of the ad ID.

[0065] By optimizing the data preprocessing process, the efficiency and accuracy of model prediction are improved compared with manual processing.

[0066] S103: Standardize based on preset rules for unifying the input format, then obtain the preprocessed data based on the normalized result, input it into the prediction model for training and testing, and output the prediction result according to the set path.

[0067] Specifically, in order to support flexible replacement of multiple models, a unified input format is required. In one embodiment, standardized input rules (specifications) are defined.

[0068]

[0069] Among them, train_file_path is the training set file path, test_file_path is the test set file path, remove_cols is the first target column, and target_cols is the second target column.

[0070] Based on this rule, a standardized result is obtained, which includes a training set file path, a test set file path, a first target column, and a second target column. The training set file path allows you to obtain training data (corresponding to the training set file) from the path and input it into the model for training; the test set file path allows you to obtain test data (corresponding to the test set file) from the path and input it into the model for testing; the first target column allows you to obtain the columns that need to be deleted, and based on the first target column, you can avoid interference from irrelevant columns in model training and testing; the second target column allows you to obtain the columns that need to be predicted, and based on the second target column, you can clearly identify the objects that need to be predicted.

[0071] Through this structured input definition, it is stipulated that the input must contain these four fields. Therefore, when replacing the model, only the definition of the model needs to be changed, without changing the input part, to achieve seamless access to the model, thereby optimizing the adaptability of the model.

[0072] It should be noted that if the file column does not contain the second target column, or there is an undeclared column, an error prompt will be output.

[0073] By defining an abstract base class (defining an interface), all models can be forced to implement unified training, prediction, and result saving methods, thereby providing a common interface for the training and testing of subsequent machine learning models. The model's input parameters include: the training set file path, the test set file path, the first target column name list (i.e., the first target column), the second target column name list (i.e., the second target column), the directory where the model training results (such as weights and evaluation reports) are saved, the start date of the training set date range, the end date of the training set date range, the actual date identifier corresponding to the test data, the feature column name list used by the model, and the output file path for the test set prediction results and evaluation indicators. The output file path for the test set results and evaluation indicators, and the directory where the model training results are saved, constitute the prediction result output directory. This prediction result output directory is used as the set path, and the prediction results are output (stored) according to the set path.

[0074] In this embodiment, the prediction results include the predicted value, the true value, the conclusion file, the predicted value and the true value comparison result file and the model metadata file. Specifically, the output structure of the model is as follows:

[0075]

[0076] Among them, output_dir is the output directory, and conclusion.txt (i.e., conclusion file) is used to record model evaluation parameters (e.g., MSE = 0.25, R 2 =0.89), ad_estimated_commission_test_results.txt (i.e., the predicted value and true value comparison result file) is used to record the comparison results of the predicted values ​​and true values ​​of the column alignment; model metadata.json (i.e., the model metadata file) is used to record the model type and hyperparameters (for example, {"model_type":"XGBoost","n_estimators":100}).

[0077] In this embodiment, the model evaluation parameters include MSE and R 2 Among them, MSE (Mean Squared Error) is used to measure the average square difference between the predicted value and the true value. The smaller the MSE value, the more accurate the model's prediction. 2 A measure of how well the model explains the variability in the data. 2 The closer the value is to 1, the better the model fits the data and can explain most of the variation.

[0078] By recording model evaluation parameters, relevant personnel can intuitively understand the performance of various machine learning models, and thus select the optimal machine learning model or build an integrated model based on actual needs.

[0079] In one embodiment, the method of the present invention further includes: selecting the optimal prediction model from the candidate prediction models based on the model evaluation parameters. First, the machine learning model to be used (such as a random forest model, a support vector machine model, a gradient boosting tree model, a neural network model, etc.) is selected, and then the performance parameters of these models (i.e., model evaluation parameters) are obtained in turn. Based on the performance parameters of the models, the optimal prediction model is selected from these models according to actual needs. For example, the prediction model with the largest MSE value can be selected as the optimal prediction model, or the optimal prediction model can be selected by comprehensively considering the model evaluation parameters (for example, by weighted summation).

[0080] Furthermore, the method of the present invention further includes: after sorting the test results of the test set by true values, performing stratified sampling according to a preset stratified sampling strategy, and then drawing and outputting a comparison chart of the predicted values ​​and the true values. Specifically, the test results are sorted in descending order by true values ​​(if the true values ​​are the same, they are sorted in descending order by predicted values), stratified sampling is performed based on the sorting results, and then a comparison chart of the predicted values ​​and the true values ​​is drawn.

[0081] In one embodiment, the stratified sampling strategy is:

[0082]

[0083] The comparison chart of the predicted values ​​and true values ​​obtained through the above-mentioned stratified sampling strategy can intuitively show the performance of the prediction model on different data segments, thereby helping relevant personnel to quickly evaluate the prediction accuracy and potential bias of the model, so that timely adjustments can be made to ensure the flexibility and adaptability of the prediction task.

[0084] In addition, the method of the present invention may further include: generating optimization suggestions based on the prediction results and the visualization results, for example, adjusting the advertising delivery strategy in the DSP report in combination with the prediction results.

[0085] It should be noted that the method of the present invention is not only capable of predicting order commissions, but can also be used for prediction tasks in e-commerce, finance, and other fields, such as sales forecasting and credit risk assessment. Specifically, by adjusting the acquisition path and the target column (the second target column, or the model's input parameter) definition, it can adapt to data structures in different fields, thereby achieving multi-dimensional predictions beyond commissions.

[0086] In summary, the present invention provides a method for optimizing a prediction model, including standardization, automated data downloading, preprocessing, and input and output processes, which provides a universal interface for subsequent machine learning model training. The universal interface can achieve cross-scenario reuse and support seamless access to multiple machine learning models, significantly improving the accuracy and efficiency of prediction model predictions.

[0087] Figure 2 FIG. 1 is a block diagram schematically illustrating a structural diagram of an optimization system for a prediction model for a DSP platform according to this embodiment.

[0088] In a second aspect, the present invention also provides an optimization system for a prediction model of a DSP platform. Figure 2 As shown, the optimization system includes a processor and a memory, wherein the memory stores computer program instructions. When the computer program instructions are executed by the processor, an optimization method for a prediction model for a DSP platform according to the first aspect of the present invention is implemented.

[0089] The optimization system also includes other components well known to those skilled in the art, such as a communication interface. The configuration and functions of these components are known in the art and will not be described in detail here.

[0090] In the present invention, the aforementioned memory can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. For example, the computer-readable storage medium can be any suitable magnetic storage medium or magneto-optical storage medium, such as resistive random access memory (RRAM), dynamic random access memory (DRAM), static random access memory (SRAM), enhanced dynamic random access memory (EDRAM), high-bandwidth memory (HBM), hybrid memory cube (HMC), etc., or any other medium that can be used to store the required information and can be accessed by an application, module, or both. Any such computer storage medium can be part of a device or accessible or connectable to a device. Any application or module described in the present invention can be implemented using computer-readable / executable instructions that can be stored or otherwise retained by such a computer-readable medium.

[0091] While several embodiments of the present invention have been shown and described herein, it will be apparent to those skilled in the art that such embodiments are provided by way of example only. Numerous modifications, variations, and alternatives will occur to those skilled in the art without departing from the concept and spirit of the present invention. It should be understood that various alternatives to the embodiments of the present invention described herein may be employed in practicing the present invention.

Claims

1. A method for optimizing a prediction model for a DSP platform, characterized in that: include: Create an experiment directory; Based on the data synchronization protocol, sample data is obtained from a preset pre-stored data directory or a remote database to the experimental directory; Preprocessing the target files in the experimental directory to obtain preprocessed data; Standardization is performed based on preset rules for unifying the input format, and then the corresponding preprocessed data is obtained based on the standardized results, and input into the prediction model for training and testing, and the prediction results are output according to the set path; the prediction results include predicted values ​​and true values, and the prediction model is a machine learning model.

2. The optimization method for the prediction model of the DSP platform according to claim 1, characterized in that: Creating an experiment directory includes: creating the experiment directory according to the experiment root directory path and the experiment identifier.

3. The optimization method for the prediction model of the DSP platform according to claim 1, characterized in that: Acquiring sample data from a preset pre-stored data directory or a remote database to the experimental directory includes: Obtain a date range, where the date range includes a training set date range and a test set date range; For any target date in the date range, determine whether there is a sample data file corresponding to the target date in the pre-stored data directory; if so, copy the sample data file from the pre-stored data directory to the training set file directory or the test set file directory under the experimental directory; if not, pull sample data from the remote database according to the first granularity of the target date to obtain a first-granularity sample data file; aggregate all first-granularity sample data files corresponding to the target date into a second-granularity sample data file, and store it in the training set file directory or the test set file directory under the experimental directory.

4. The optimization method for the prediction model of the DSP platform according to claim 3, characterized in that: Preprocess the target files in the experimental directory, including: For the files in the training set file directory, all files are spliced ​​together to obtain the training set file; for the files in the test set file directory, all files are spliced ​​together to obtain the test set file; The first target column in the training set file and the test set file is deleted, and the second target column with missing values ​​is filled.

5. The optimization method for the prediction model of the DSP platform according to claim 1, characterized in that: The standardized result includes a training set file path, a test set file path, a first target column, and a second target column. Acquiring corresponding preprocessed data based on the standardized result includes: Acquire test data according to the test set file path; acquire training data according to the training set file path; acquire columns to be deleted according to the first target column, and acquire columns to be predicted according to the second target column.

6. The optimization method for the prediction model of the DSP platform according to claim 1, characterized in that: Before creating the experiment directory, also include: clearing the visualization result directory and the raw data directory.

7. The optimization method for the prediction model of the DSP platform according to claim 1, characterized in that: The prediction results also include a conclusion file for recording model evaluation parameters, and a model metadata file for recording model type and hyperparameters; The optimization method further includes: selecting an optimal prediction model from candidate prediction models according to the model evaluation parameters recorded in the conclusion file.

8. The optimization method for the prediction model of the DSP platform according to claim 1, characterized in that: Outputting the prediction result according to the set path includes: determining the set path according to input parameters of the input prediction model, and the input parameters include the prediction result output directory.

9. The optimization method for the prediction model of the DSP platform according to claim 1, characterized in that: Also includes: After sorting the test results of the test set by the true values, stratified sampling is performed according to the preset stratified sampling strategy, and then a comparison chart between the predicted values ​​and the true values ​​is drawn and output.

10. An optimization system for a prediction model on a DSP platform, characterized in that: The invention comprises a processor and a memory, wherein the memory stores computer program instructions, and when the computer program instructions are executed by the processor, the optimization method for the prediction model for the DSP platform according to any one of claims 1 to 9 is implemented.

Citation Information

Patent Citations

  • Format conversion device and method, and computer program

    JP2004341605A