A model training method and device, equipment and storage medium
By performing operations such as data merging, outlier handling, null value imputation, resampling, and data completion on the target dataset, the problem of insufficient dataset processing in artificial intelligence development frameworks is solved, and automated dataset processing and training configuration are realized, thereby improving dataset quality and adaptability.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- CONTEMPORARY AMPEREX FUTURE ENERGY RES INST (SHANGHAI) LTD
- Filing Date
- 2024-12-25
- Publication Date
- 2026-06-26
AI Technical Summary
Existing AI development frameworks cannot provide users with dataset processing capabilities, resulting in low dataset quality that fails to meet model training requirements and cannot be used as training samples.
This paper provides a model training method that, by acquiring the target data processing method configured by the user, performs operations such as data merging, outlier handling, null value imputation, resampling, and data completion on the target dataset, constructs training instances, and executes training, thereby realizing automated processing and configuration of the dataset.
It achieves automated processing of datasets, meets model training requirements, enables datasets to be used as samples in training, improves the quality and adaptability of datasets, and simplifies the user configuration process.
Smart Images
Figure CN122286288A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of artificial intelligence technology, and in particular to a model training method, apparatus, device, and storage medium. Background Technology
[0002] With the rapid development of deep learning technology, neural network models have gradually become an important tool in the field of artificial intelligence and are widely used in production and life. For example, models can be used for parameter prediction. Through model training, neural network models can learn the inherent laws and patterns of data, thereby enabling the prediction of future parameters.
[0003] Users can use AI development frameworks to train models, but current AI development frameworks cannot provide users with dataset processing capabilities, and datasets that do not meet training requirements cannot be used. Summary of the Invention
[0004] This application provides at least one model training method, apparatus, device, and storage medium.
[0005] This application provides a model training method, comprising: obtaining a user-configured target data processing method, and processing the target dataset according to the target data processing method, wherein the target data processing method includes several combined processing methods and / or several single processing methods, the combined processing method includes sequentially executing the processing corresponding to multiple single processing methods, and the single processing method includes at least one of data merging, outlier handling, null value imputation, resampling, and data completion; using the data-processed target dataset as the sample dataset for training instances corresponding to the target model; and executing training instances to train the target model using the target dataset and obtain training results.
[0006] In the above scheme, the target dataset can be automatically processed according to the user configuration, so that the target dataset can meet the model training requirements and participate in the training as a sample dataset for the target model. The data processing configuration function of the dataset is realized, so that the user can conveniently configure the data processing to be performed on the dataset, and the device automatically performs the corresponding processing of the dataset according to the user configuration.
[0007] In some embodiments, obtaining the target data processing method configured by the user and processing the target dataset according to the target data processing method includes: displaying a combined processing method selection page, displaying at least one preset combined processing method for the user to select; in response to the user's combined selection operation, selecting several combined processing methods from the at least one preset combined processing method and performing processing corresponding to the several combined processing methods on the target dataset; in response to ending the combined selection operation, displaying a single processing method selection page, displaying at least one preset single processing method for the user to select; in response to a single selection operation, selecting several single processing methods from the at least one preset single processing method and performing processing corresponding to the several single processing methods on the target dataset.
[0008] In the above scheme, users are provided with selection pages for combined processing methods and single processing methods, so that users can configure the target processing method of the dataset and realize the automated processing of the dataset.
[0009] In some embodiments, the target dataset includes multiple data entries, each containing multiple fields; the multiple data entries are sorted by time; data merging includes: merging multiple data entries to be merged into one integrated data entry, where the times in the multiple data entries to be merged are the same; the values of each field in the integrated data entry are determined based on the values of the corresponding fields in the multiple data entries to be merged; outlier handling includes: selecting an anomaly judgment field from multiple fields based on the processing configuration information corresponding to the target dataset; replacing the values of the anomaly judgment fields that do not meet the preset field value requirements with null values; null value imputation includes: using fields with null values as fields to be filled, finding the corresponding fill values for the fields to be filled from the multiple data entries based on the processing configuration information corresponding to the target dataset, and using the fill values to fill the fields to be filled; resampling includes: determining a preset sampling frequency based on the processing configuration information corresponding to the target dataset; resampling multiple data entries according to the preset sampling frequency to obtain multiple resampled data entries; data completion includes: determining a preset time interval based on the processing configuration information corresponding to the target dataset; dividing the multiple data entries into several data groups according to the preset time interval; sequentially using the data groups with data gaps as target data groups, and filling the target data groups with data from the preceding data group.
[0010] The above solution provides functions such as data merging, outlier handling, null value imputation, resampling, and data completion, which can be flexibly applied to the processing of the target dataset.
[0011] In some embodiments, before using the processed target dataset as the sample dataset for training instances corresponding to the target model, the method further includes: in response to a model selection operation, determining a target model from at least one preset model and determining the current parameter configuration information of the target model; and in response to a function selection operation, determining a target loss function from at least one preset loss function; constructing training instances using the target model, the current parameter configuration information, and the target loss function, wherein when the training instances are executed, the target model is trained based on the target loss function.
[0012] The above solution provides a model and loss function selection function, so that users can configure the model to be trained and the loss function to be used, automatically build the corresponding training instance, and realize the automated training of the target model.
[0013] In some embodiments, constructing a training instance using the target model, current parameter configuration information, and target loss function includes: using the target model, current parameter configuration information, and target loss function as attribute values corresponding to the wrapper class; executing the class constructor method of the wrapper class to construct the training instance.
[0014] In the above scheme, the user-configured model, parameters, and loss function are used as attribute values, and the training instances are automatically constructed using a class construction method.
[0015] In some embodiments, determining the current parameter configuration information of the target model includes: obtaining the pre-stored parameter configuration information of the target model; and, in response to the user's parameter input operation, obtaining the user-inputted temporary parameter configuration information; and obtaining the current parameter configuration information of the target model based on the temporary parameter configuration information and the pre-stored parameter configuration information.
[0016] The above scheme provides a parameter input function so that users can configure the temporary parameters to be used. It can automatically determine the parameters used for training the target model by combining the user-configured temporary parameters and the pre-stored parameters.
[0017] In some embodiments, the method further includes: in response to a training resource creation instruction, obtaining training resource attributes input by a user, obtaining a training resource file based on the training resource attributes, and storing the training resource attributes and the training resource file, wherein the training resource attributes and the training resource file are used to define a preset training resource; wherein the preset training resource includes at least one of a preset model, a preset loss function, and a preset dataset, and the preset dataset is used for the user to select as a target dataset.
[0018] The above solution provides a training resource creation function, which allows users to create preset training resources that can be reused in subsequent training processes, thereby improving the efficiency of model training configuration.
[0019] In some embodiments, the training resource attributes include the original storage path, and obtaining the training resource file based on the training resource attributes includes: obtaining the training resource file from the original storage path.
[0020] In the above scheme, the training resource file can be obtained according to the original storage path configured by the user, so that the preset training resource can be used for training in the future, thereby realizing the automation of model training.
[0021] In some embodiments, storing training resource attributes and training resource files includes: in response to the training resource attributes including a resource storage method, constructing a custom storage interface based on the resource storage method, and calling the custom storage interface to store the training resource attributes and training resource files; in response to the training resource attributes not including a resource storage method, obtaining a target storage address from the training resource attributes, and storing the training resource attributes and training resource files at the target storage address.
[0022] The above solution provides a custom storage function, which can customize storage or store data to a specified address according to user needs, adapting to different application scenarios.
[0023] In some embodiments, the training resource attributes include a unique identifier attribute; the method further includes: in response to a training resource update instruction, obtaining an update attribute and a unique identifier attribute input by the user; in response to the existence of a preset training resource corresponding to the input unique identifier attribute, updating the currently stored corresponding training resource attribute using the update attribute; in response to the absence of a preset training resource corresponding to the input unique identifier attribute, indicating update failure.
[0024] The above solution provides a training resource update function, which can update the attributes of existing preset training resources and meet the user's need to change preset training resources.
[0025] In some embodiments, the training results include a first model obtained from the last training and a second model with the minimum loss; the method further includes: in response to a user's inference model selection operation, selecting an inference model from the first model and the second model; and using the inference model to perform parameter prediction based on the data to be processed to obtain a prediction result.
[0026] The above scheme provides a way to retain two models as training results, so that users can flexibly choose the model that is more suitable for their needs.
[0027] In some embodiments, the combined processing methods include a target combined processing method; the target combined processing method includes sequentially performing data merging, outlier handling, null value imputation, resampling, null value imputation, data completion, and null value imputation.
[0028] In the above scheme, a specific combination of data processing methods is used to process the target dataset. Using target combination processing can improve the data quality of the target dataset, making the target dataset more suitable for the training needs of the target model.
[0029] This application provides a model training apparatus, including a data processing module, a sample determination module, and an execution module. The data processing module acquires a user-configured target data processing method and processes the target dataset according to that method. The target data processing method includes several combined processing methods and / or several single processing methods. Combined processing methods include sequentially executing multiple single processing methods. Single processing methods include at least one of data merging, outlier handling, null value imputation, resampling, and data completion. The sample determination module uses the processed target dataset as the sample dataset for training instances corresponding to the target model. The execution module executes the training instances to train the target model using the target dataset and obtain training results.
[0030] This application provides an electronic device, including a memory and a processor, wherein the memory stores program instructions, and the program instructions are executed by the processor to implement the above-described method.
[0031] This application provides a computer-readable storage medium having program instructions stored thereon, which, when executed by a processor, implement the above-described method.
[0032] In the above scheme, the target dataset can be automatically processed according to the user configuration, so that the target dataset can meet the model training requirements and participate in the training as a sample dataset for the target model. The data processing configuration function of the dataset is realized, so that the user can conveniently configure the data processing to be performed on the dataset, and the device automatically performs the corresponding processing of the dataset according to the user configuration.
[0033] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this application. Attached Figure Description
[0034] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the specification, serve to explain the technical solutions of this application.
[0035] Figure 1 This is a flowchart illustrating the model training method provided in some embodiments of this application;
[0036] Figure 2 This is a flowchart illustrating a model training method provided in other embodiments of this application;
[0037] Figure 3 This is a flowchart illustrating a model training method provided in some embodiments of this application;
[0038] Figure 4 This is a schematic diagram of a model training method provided in some embodiments of this application;
[0039] Figure 5 This is a schematic diagram of the framework of a model training device provided in some embodiments of this application;
[0040] Figure 6 This is a schematic diagram of the framework of an electronic device provided in some embodiments of this application;
[0041] Figure 7 This is a schematic diagram of the framework of a computer-readable storage medium provided in some embodiments of this application. Detailed Implementation
[0042] The embodiments of this application will now be described in detail with reference to the accompanying drawings.
[0043] In the following description, specific details such as particular subsystem structures, interfaces, and technologies are presented for illustrative purposes rather than for limiting purposes, in order to provide a thorough understanding of this application.
[0044] In this document, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the character " / " generally indicates that the preceding and following related objects have an "or" relationship. Furthermore, "many" in this document means two or more. Moreover, the term "at least one" in this document means any combination of at least two of any one or more of a plurality of objects. For example, including at least one of A, B, and C can mean including any one or more elements selected from the set consisting of A, B, and C.
[0045] Users can train models using AI development frameworks; however, current AI development frameworks lack dataset processing capabilities. In some cases, datasets may have low quality, failing to meet model training requirements and thus unusable as training samples. Therefore, this embodiment proposes the following model training method, which automatically processes the target dataset according to user configuration, ensuring the target dataset meets model training requirements and can be used as a training sample dataset for the target model. This method implements dataset processing configuration functionality, allowing users to easily configure the necessary data processing for the dataset. The device then automates dataset processing according to the configuration, facilitating full utilization of the dataset.
[0046] The model training method disclosed in this application can be applied to the training process of neural network models.
[0047] See Figure 1 , Figure 1 This is a flowchart illustrating a model training method provided in some embodiments of this application. Specifically, the method includes:
[0048] Step S110: Obtain the target data processing method configured by the user, and process the target dataset according to the target data processing method.
[0049] The target data processing methods include several combined processing methods and / or several single processing methods.
[0050] In some embodiments, the single processing method includes at least one of data merging, outlier handling, null value imputation, resampling, and data completion. The single processing method can also be adjusted according to the data processing needs in the actual scenario.
[0051] It is understandable that a combined processing method means executing the processes corresponding to multiple individual processing methods sequentially. Different combined processing methods contain different individual processing methods. A single processing method can appear once or multiple times in a combined processing method.
[0052] In some embodiments, the processing of the target dataset may include a target combination processing method, which may include sequentially performing data merging, outlier handling, null value imputation, resampling, null value imputation, data completion, and null value imputation.
[0053] In some embodiments, the device can provide users with data processing configuration functions. Specifically, the processing method of the target dataset can be configured by providing selection, input, or other methods.
[0054] In some implementation scenarios, the device can provide users with preset data processing methods for selection to achieve the configured target data processing method. Specifically, the device can pre-store several preset single processing methods and / or several preset combinations of processing methods for users to choose from.
[0055] In some cases, the device can also provide alternative configuration options for the target data processing method, such as providing user input for configuring the target data processing method. The specific configuration can be tailored to the specific application requirements.
[0056] Step S120: Use the target dataset after data processing as the sample dataset for training instances corresponding to the target model.
[0057] The training instances corresponding to the target model can be used to train the target model. Besides using training instances, model training also requires determining the sample data to be used; the target dataset, after data processing, can be used as the sample dataset.
[0058] Step S130: Execute a training instance to train the target model using the target dataset and obtain the training results.
[0059] In some embodiments, the target model trained by the model training method provided in this application can be used for parameter prediction. The parameters to be predicted can be set according to actual application needs, and the sample data or inference data used can include data related to the parameters to be predicted. For example, the target model can be used to predict photovoltaic power generation related parameters, such as photovoltaic power generation efficiency, and the sample data can include temperature, photovoltaic power, parameters characterizing air conditions, etc., related to photovoltaic power generation efficiency.
[0060] In the above solution, the target dataset can be automatically processed according to the user configuration, so that the target dataset can meet the model training requirements and participate in the training as a sample dataset for the target model. The data processing configuration function of the dataset is realized, which allows the user to conveniently configure the data processing to be performed on the dataset. The device automatically performs the corresponding processing of the dataset according to the user configuration, making full use of the dataset.
[0061] In some embodiments, the device can provide users with preset data processing methods for selection to configure the target data processing method. Furthermore, all preset data processing methods can be displayed for users to view and select; alternatively, the pre-stored data processing methods may include at least one preset combined processing method and at least one preset single processing method, which can be displayed separately for users to view and select.
[0062] In some implementation scenarios, the device can first provide preset combined processing methods for selection, and then provide preset single processing methods for selection. Specifically, the device displays a combined processing method selection page, which shows at least one preset combined processing method for the user to choose from. In response to the user's combined selection operation, several combined processing methods are selected from the at least one preset combined processing method, and the processing corresponding to the several combined processing methods is performed on the target dataset. In response to the end of the combined selection operation, a single processing method selection page is displayed, which shows at least one preset single processing method for the user to choose from. In response to the user's single selection operation, several single processing methods are selected from the at least one preset single processing method, and the processing corresponding to the several single processing methods is performed on the target dataset.
[0063] A single selection operation can choose one or more processing methods. If multiple processing methods are selected in a single operation, the user can be prompted to confirm the order in which the selected processing methods are chosen. One or more selection operations can be performed on the selection page. The device can execute the corresponding processing method for each selection operation made by the user.
[0064] In some embodiments, the model training method provided in this application involves data including multiple fields, and each data entry may correspond to a time period. Multiple data entries in the target dataset can be arranged according to time.
[0065] In some implementation scenarios, the target dataset contains multiple fields, including time. All data is arranged in chronological order. The values of each field in the above data are specific numerical values.
[0066] In some embodiments, the data merging process provided in this application may include selecting multiple data entries with the same time frame as data to be merged, and merging these multiple data entries into a single integrated data entry. The values of each field in the integrated data entry are determined based on the values of the corresponding fields in the data to be merged.
[0067] Specifically, the fields contained in the integrated data are identical to those in the original data to be merged. The value of a field in the integrated data is determined based on the values of that field in all the data to be merged. For example, if the value of field A is 'a' in all the data to be merged, the value of field A in the integrated data is 'a'. If the values of field B in all the data to be merged are b1, b2, and b3, the value of field B in the integrated data can be determined based on these three values, for example, by selecting from these three values or by calculating them.
[0068] In some embodiments, the outlier handling provided by this application may include: selecting an outlier judgment field from multiple fields based on the processing configuration information corresponding to the target dataset, and replacing the value of the outlier judgment field that does not meet the preset field value requirements with an empty value.
[0069] The preset field value requirement is the requirement for the value of the exception judgment field. If the value of the exception judgment field in a piece of data does not meet the preset field requirement, it can be considered an exception value, and the exception value will be adjusted to an empty value.
[0070] In a specific application scenario, the anomaly detection field is the state of charge, with a preset value requirement of 0 to 100. The state of charge field is checked across all data; if the value does not meet the requirement of being between 0 and 100, it is set to null.
[0071] The target dataset includes corresponding processing configuration information, which can be configured by the user. This configuration information contains settings related to outlier handling, specifically outlier judgment fields and preset field value requirements. There can be one or more outlier judgment fields, each with its own preset field value requirements. For each outlier judgment field, the outlier value is set to null according to its preset value requirements. For example, the outlier judgment fields include field A and field B. The preset field value requirement for field A is a value between 0 and 100, and the preset field value requirement for field B is a value of 0 or 1.
[0072] In some embodiments, the null value filling provided by this application may include: taking a field with an empty value in a data as a field to be filled, determining the filling value corresponding to the field to be filled from multiple data based on the processing configuration information corresponding to the target dataset, and filling the field to be filled with the filling value.
[0073] The configuration information can include settings related to null value filling, indicating how to find a fill value for a field to be filled. Specifically, values from the same field in other data can be used as fill values.
[0074] In some implementation scenarios, the filling method for a field to be filled can be forward filling, backward filling, or linear fitting filling. Forward filling refers to filling missing values based on prior non-missing values, backward filling refers to filling missing values based on subsequent non-missing values, and linear fitting filling refers to filling missing values using a linear interpolation method.
[0075] In some implementation scenarios, a value range can be set for the above filling methods to indicate the range of values to be filled. For example, for the state of charge field, using forward filling with a value range of 5 indicates that the search proceeds backward from the field to be filled, searching for at most the first 5 data entries. Another example is to define a value range for linear fitting filling, using valid values within the range for linear fitting. It is understandable that different null value filling logic can be configured for different fields.
[0076] In some embodiments, the resampling process provided in this application may include: determining a preset sampling frequency based on the processing configuration information corresponding to the target dataset, and resampling multiple data points according to the preset sampling frequency to obtain multiple resampled data points.
[0077] In some implementation scenarios, multiple data points in the target dataset can be sorted by time. If the sampling frequency of the current multiple data points differs from the configured preset sampling frequency, resampling can be performed. Specifically, resampling can be achieved through interpolation or averaging. If the data sampling frequency is uneven, resampling can also be performed to achieve a uniform sampling frequency.
[0078] Understandably, multiple data points in the target dataset can be sorted chronologically. Resampling can be applied to a subset of the target dataset; for example, if there is uneven sampling within certain time periods, then the data within those time periods will be resampled. If the sampling frequency meets the requirements for some time periods, then no processing is necessary. In some cases, if data is missing in certain time periods, resampling may also be unnecessary to avoid introducing unreasonable resampling results.
[0079] In a specific application scenario, multiple data points in the target dataset are obtained by sampling once every 10 seconds. If the preset sampling frequency is once every 30 seconds, then the data in the target dataset can be processed by sampling mean to obtain resampled data.
[0080] In some embodiments, resampling can be performed by calculating central tendency statistics on existing data, such as the average.
[0081] In some embodiments, the data completion processing provided in this application may include: determining a preset time interval based on the processing configuration information corresponding to the target dataset, dividing multiple data into several data groups according to the preset time interval, taking the data group with data gaps as the target data group, and filling the target data with data from other data groups.
[0082] This means that the above data completion process can be applied to datasets with large amounts of missing data. Furthermore, the data groups with missing data can be sequentially designated as target data groups, and the data from the preceding data group can be used to fill the target data group.
[0083] The target data set can be either completely missing or partially missing. If it is completely missing, data from the previous data set can be directly added to the target data set. If it is partially missing, a corresponding number of data points can be selected from the previous data set and added to the target data set.
[0084] The preset time interval can be week, day, or hour. In a specific application scenario, multiple data entries are divided into several data groups according to days. The data groups are arranged in chronological order, and data groups with missing data are selected as target data groups, filling them with data from the preceding data group.
[0085] In some specific application scenarios, the device can provide users with several preset combined processing methods and several preset single processing methods. The preset single processing methods can include at least one of data merging, outlier handling, null value imputation, resampling, and data completion. The preset combined processing methods can include a target combined processing method, which can include performing data merging, outlier handling, null value imputation, resampling, null value imputation, data completion, and null value imputation in sequence.
[0086] In the above combined processing method, data merging is first performed to eliminate duplicate data, reducing repetitive processing steps for duplicate data and preventing duplicate data from affecting the effectiveness of subsequent processing steps. Next, outliers are handled to prevent them from filling other fields and affecting data quality. Then, null value imputation is performed to fill individual field values to improve data integrity and facilitate resampling. After resampling, null value imputation is performed again; at this point, the data meets the sampling frequency requirements, avoiding the introduction of unreasonable filler values. After implementing sampling frequency adjustment and individual field null value imputation, existing field values are fully and rationally utilized to improve data quality. Then, for data missing within a certain period, data completion is used to further improve the completeness of the dataset. Null value imputation is performed on this basis, fully utilizing the results of data completion and referencing data from other time periods to further adjust individual field values, achieving further improvement in dataset quality. This combined processing method is generally applicable to datasets arranged by time, providing users with an optional combined processing method and improving the convenience of configuring dataset processing methods.
[0087] Of course, the preset combined processing method is not limited to the above target combined processing method. For example, another preset combined processing method can be to perform data merging, outlier handling, null value filling, resampling, and data completion in sequence.
[0088] It should be noted that for datasets with a chronological order, the system provides users with a general data processing approach by offering preset target combination processing methods. This allows users to directly select this method to process the dataset, effectively improving data quality, reducing anomalies, duplications, null values, and missing values. Furthermore, the entire processing process does not introduce values that differ significantly from the actual data, thus avoiding negative impacts on subsequent training.
[0089] In some embodiments, the device can also provide users with model configuration functionality, allowing the device to determine the target model based on the user's configuration operations before training the target model.
[0090] In some implementation scenarios, the device may store at least one preset model, display at least one preset model for the user to select, and determine the target model from at least one preset model in response to the model selection operation.
[0091] In some cases, other ways to configure the model can also be provided, such as configuring the model used for this training by inputting or uploading files.
[0092] In some embodiments, after the target model is determined, the current parameter configuration information used for this training can also be determined for the target model.
[0093] In some embodiments, the device can provide users with model parameter configuration functions, and before training, it can also determine the current parameter configuration information used by the target model for this training based on the user's configuration operations on the model.
[0094] In some cases, the selected preset model may have corresponding default parameter configuration information, which can be used as the current parameter configuration information.
[0095] In some implementation scenarios, the device can provide users with a parameter input page so that users can configure parameters by inputting them.
[0096] In some implementation scenarios, the preset model can have at least one pre-stored parameter configuration information available for user selection. Further, the current parameter configuration information used by the target model for this training can be obtained based on the selected pre-stored parameter configuration information and the temporary parameter configuration information input by the user. Further, the temporary parameter configuration information input by the user can include the complete parameter configuration information required by the model, or only a portion of the parameter configuration information. Between the temporary parameter configuration information and the selected pre-stored parameter configuration information, the configuration information in the temporary parameter configuration information is used first; for parts not included in the temporary parameter configuration information, the corresponding parts from the selected pre-stored parameter configuration information are used.
[0097] In a specific application scenario, one of the pre-stored parameter configuration information can be used as the default parameter configuration information. If the user does not select pre-stored parameter configuration information, the default parameter configuration information will be used as the selected parameter configuration information.
[0098] In a specific application scenario, if the user does not input temporary parameter configuration information, the selected pre-stored parameter configuration information can be used directly as the current parameter configuration information.
[0099] In a specific application scenario, temporary parameter configuration information input by the user is obtained, and the current parameter configuration information is determined based on the temporary parameter configuration information and the default parameter configuration information.
[0100] In some embodiments, a loss function is used when training the target model, and the device can also provide users with a loss function configuration function to determine the target loss function based on the user's configuration operation of the loss function.
[0101] In some cases, other configuration functions can also be provided, such as providing input methods to configure the loss function.
[0102] In some implementation scenarios, the device can store at least one preset loss function for the user to select. In response to the function selection operation, the target loss function is determined from at least one preset loss function.
[0103] In some embodiments, after determining the target model, current parameter configuration information, and target loss function, these three elements can be used to construct a training instance of the target model. When the training instance is executed, it trains the target model using the current parameter configuration information to obtain the training result, wherein the target loss function is used as the loss function for training.
[0104] In some embodiments, the wrapper class is a class that can be used to implement model training. The target model is used as an attribute value of the wrapper class to construct a corresponding training instance, enabling the training instance to train the target model. The wrapper class can quickly construct a model training instance that matches the user's needs based on user configuration, and the training instance can achieve automated model training.
[0105] In some embodiments, the target model, current parameter configuration information, and target loss function are used as attribute values of the wrapper class. The class constructor of the wrapper class is then executed to construct a training instance. The training instance is an instance of the wrapper class. When the methods of the training instance are executed, they train the target model with the current parameter configuration information based on the target loss function.
[0106] In some embodiments, at least one of the model, loss function, and dataset serves as a training resource. The device can provide users with management functions for training resources. For example, it can provide users with functions such as adding, modifying, and viewing training resources.
[0107] Understandably, users can add, modify, and view preset training resources, which are available for selection when configuring model training. During model training configuration, the device stores preset training resources that users can choose from; these preset training resources may be built and / or modified in response to user actions.
[0108] In some implementation scenarios, the device can respond to user commands for managing training resources, such as creating new preset training resources, modifying preset training resources, and viewing preset training resources.
[0109] In some embodiments, in response to a resource creation command, the system obtains training resource attributes input by the user, retrieves a training resource file based on the training resource attributes, and stores the training resource attributes and the training resource file. The training resource attributes and the training resource file can be used to define a preset training resource; storing both allows the construction of the preset training resource.
[0110] The preset training resources include at least one of a preset model, a preset loss function, and a preset dataset. After creating the preset training resources, they can be provided to the user for configuration during any subsequent model training process. The preset model, preset loss function, and preset dataset can all be selected by the user as the target model and target loss function, respectively.
[0111] Of course, in some cases, users can configure the system without selecting preset training resources, and can choose other methods, such as configuring by inputting information or uploading files.
[0112] In some embodiments, training resource attributes may include an original storage path, which may indicate the original storage location of the training resource files. This allows the device to retrieve the training resource files from the original storage path.
[0113] In some embodiments, training resource attributes may include resource storage methods, which can be used to customize the storage method of preset training resources.
[0114] In some implementation scenarios, the device can store preset training resources according to the default storage method. When the user-configured training resource attributes include the resource storage method, the user-defined resource storage method will be given priority.
[0115] In some implementation scenarios, in response to training resource attributes including resource storage methods, the resources are stored according to those methods. Specifically, a custom storage interface can be built based on this storage method, and the device can call this custom storage interface to store training resource attributes and training resource files.
[0116] In some implementation scenarios, preset training resources include a unique identifier attribute. This identifier can be used to uniquely identify the training resource and to manage it. For example, it can be used to identify preset training resources that are to be modified or viewed.
[0117] Understandably, if there is no preset training resource corresponding to the unique identifier attribute of the input, the device cannot find the resource that needs to be updated and its attributes, and therefore cannot perform the update.
[0118] In a specific application scenario, the attributes of the preset training model may include: target storage path (model_file_path), original storage path (source_path), model class name (class_name), model name (model_name), default parameters of the model (model_config), and model storage implementation method (model_store).
[0119] The specific content of the preset resource attributes can be configured by the user. The target storage path indicates the storage path of the model's core Python code files; furthermore, it can be the storage path under the default storage implementation. The model storage implementation defaults to local computer storage, but custom model storage implementations are supported.
[0120] The model name can be used as a unique identifier. Alternatively, other unique identifiers can be set, such as the model code.
[0121] In a specific application scenario, the original storage path (source_path) is a required field, while other attributes can be optional.
[0122] Furthermore, when the model storage implementation is a custom method, the original storage path (source_path) is not required. When the model storage implementation is the default storage method, the original storage path (source_path) is required and is used to indicate the local computer storage address.
[0123] In a specific application scenario, the system retrieves the training resource attributes entered by the user, copies the file from the original storage path to the target storage path, and saves other optional information to the directory where the target storage path is located. In some cases, if the user specifies a custom model storage method, it will be stored according to the user's method.
[0124] In a specific application scenario, the attributes of the preset loss function may include: target storage path (loss_file_path), original storage path (source_path), loss function name (loss_name), custom parameters of the loss function (loss_config), and loss function storage implementation method (loss_store).
[0125] Among them, custom parameters for the loss function can be used to override the default parameters in the loss function.
[0126] In a specific application scenario, the attributes of a preset dataset may include: target storage path (data_file_path), original storage path (source_path), dataset name (data_name), data processing configuration information (data_config), and data storage implementation method (data_store).
[0127] The data processing configuration information may include the data processing rules corresponding to the dataset, instructing the device on how to process the dataset. For example, it may include anomaly detection fields and corresponding field value requirements.
[0128] In some embodiments, in response to a training resource update instruction, the update attribute and unique identifier attribute input by the user are obtained. The unique identifier attribute is used to identify the resource to be updated, and the update attribute represents the specific content to be updated. In response to the existence of a preset training resource corresponding to the input unique identifier attribute, the training resource attribute of the currently stored preset training resource is updated using the update attribute. In response to the absence of a preset training resource corresponding to the input unique identifier attribute, an update failure message is displayed.
[0129] In some implementation scenarios, the device can replace the currently stored corresponding attribute with the input update attribute.
[0130] In some implementation scenarios, if the update attribute includes the original file path, the device can obtain the new resource file from the new original file path entered by the user and replace the currently stored resource file.
[0131] In some implementation scenarios, if the input update attribute includes the target storage path, the device can migrate the resource attributes and resource files to the input target storage address.
[0132] It is understandable that a model file can include multiple model classes and their corresponding code. In some implementation scenarios, if the input update attribute includes a model class name, the new model class name will be used in subsequent calls, thereby executing the code corresponding to the new model class name in the model file.
[0133] In some implementation scenarios, if the updated attributes include the storage implementation method, the device can store resource attributes and resource files using the new storage implementation method.
[0134] In some embodiments, the device may respond to a query command for a unique identifier attribute by displaying to the user the resource attributes and / or resource files corresponding to that unique identifier attribute.
[0135] See Figure 2 , Figure 2 This is a flowchart illustrating the model training method provided in other embodiments of this application.
[0136] Specifically, the method may include: a user queries a model, the device responds to the model query instruction, displays the corresponding model, and identifies the model as the target model, the user selects whether to temporarily adjust the parameter configuration, the device determines whether there is temporary parameter configuration information, if not, the device uses the currently stored parameter configuration information of the target model, if, the device exists, the device combines the temporary parameter configuration information and the currently stored parameter configuration information to obtain the parameter configuration information used in this training.
[0137] Users can choose to use the new parameter configuration in this step to create a model for subsequent training without affecting the parameter configuration of the model already saved in the database. It is only a temporary replacement, and its effect is limited to the packaged model generated in this creation.
[0138] When a user queries a loss function, the system displays the corresponding loss function in response to the query command, identifies that loss function as the target loss function, binds the target loss function to the model, and creates the packaged model to be trained.
[0139] The loss function is a computational function used to measure the difference between the model's predicted values and the actual values, and it will be used in the subsequent model training process. The packaged model, also simply called the model, is a model obtained by wrapping the basic model and the loss function together.
[0140] Specifically, binding a target loss function to the model and creating the model to be trained can be achieved through the generation of training instances in the aforementioned embodiments.
[0141] See Figure 3 , Figure 3 This is a flowchart illustrating a model training method provided in some embodiments of this application.
[0142] Specifically, the method may include: a user queries a dataset; the device responds to the dataset query instruction, displays the corresponding dataset, and identifies the dataset as the target dataset; the user selects whether to use a governance pipeline; the device determines whether to select a preset general governance dataset pipeline process; if the user selects to use the pipeline, the device processes the target dataset accordingly; if the user does not use a governance pipeline or has already selected to use a governance pipeline process, the device determines whether to use other governance dataset methods; if the user selects to use a governance dataset method, the device processes the target dataset accordingly; after processing, the device returns to determine whether to continue using other governance dataset methods; if the user does not select to use other governance dataset methods, the device stops processing the target dataset and obtains a usable dataset.
[0143] In some implementation scenarios, datasets can be converted into PyTorch datasets. During training, PyTorch's GPU parallel computing capabilities can be used to train the model.
[0144] In some embodiments, the training result includes a trained model that can be used for inference. For example, a trained parameter prediction model can be used to perform parameter prediction to obtain parameter prediction results.
[0145] It is understandable that the training of the target model can be carried out in several rounds, and each round will produce the model trained in that round. The training result can include all the models trained in all rounds, or one or more of the models trained in all rounds can be selected as the training result. For example, the selection can be based on the number of training rounds or the loss.
[0146] In some implementation scenarios, the training results include at least one of the first model obtained from the last training and the second model with the minimum loss.
[0147] Furthermore, the training results include a first model and a second model. In response to the user's inference model selection operation, an inference model is selected from the first and second models. The inference model is then used to perform parameter prediction based on the data to be processed, thereby obtaining the prediction result. Parameter prediction is performed for a specific parameter, and the data to be processed can be data related to that specific parameter.
[0148] In a specific application scenario, the training results include the model file for the first model (end.pt) and the model file for the second model (bess.pt). Both model files can be used in subsequent inference.
[0149] In some cases, the current parameter configuration information used by the first and second models at the start of this training can also be saved, that is, the initial parameter configuration before the model training started, so that users can understand under what initial parameter configuration the two models were trained.
[0150] In some embodiments, the training instances may also define inference methods for parameter prediction using a first model and / or a second model. The models used for inference and the data to be processed are predefined for the inference method, thereby enabling the training of the target model and the use of the trained model for inference during the execution of the training instances.
[0151] See Figure 4 , Figure 4 This is a schematic diagram of a model training method provided in some embodiments of this application.
[0152] The model training method can be executed by a user-operated local device running an artificial intelligence development framework that provides artificial intelligence resource management and artificial intelligence training and inference functions.
[0153] Artificial intelligence resources include models, loss functions, and datasets. Artificial intelligence development frameworks can provide users with registration, query, and updating of artificial intelligence resources. Artificial intelligence resources are stored using local computer file systems or other storage types, such as MinIO (a distributed file storage solution).
[0154] The AI training and inference capabilities provide users with model training and inference configuration options, enabling automatic model training and inference based on user settings. Specifically, during the training configuration phase, it offers dataset processing configuration options, automatically processing the selected dataset according to user preferences to ensure the dataset closely matches the model training needs.
[0155] This proposal suggests a framework developed using the Python language. The framework is ultimately packaged as a Python package, requiring no additional deployment from the user; they can directly reference the package. This package also includes several commonly used AI development frameworks, enabling local training and inference of AI algorithms on the user's computer. Provided the user's device performance is sufficient, local model training and inference are achieved. Uploading model files and datasets to a server is no longer necessary; all training and inference operations can be performed locally, and the resulting models can be used directly without downloading.
[0156] This proposal breaks down the resources required for AI algorithm training into three definitions: model, loss function, and dataset, and provides interfaces for registering, updating, and querying them. Specifically, a model consists of the AI algorithm model code file, a model parameter configuration dictionary (parameter configuration information in the aforementioned embodiments), and a storage address. A loss function consists of user-developed or existing open-source loss function classes and methods, and a storage address. A dataset consists of the data files required for training and inference, a data governance configuration dictionary (data processing configuration information in the aforementioned embodiments), and a storage address. The storage address corresponds to an address on the user's computer. By defining these three resources and providing interfaces for registering, updating, and querying them, resource management for AI algorithms is achieved, providing a more efficient and complete framework for AI algorithm training and inference.
[0157] This definition, by separating files and configuration, improves efficiency by requiring only a separate configuration update when only the parameter configuration dictionary needs updating. Furthermore, since storage addresses are defined within resources, their storage location on the computer can be managed. With these interfaces, management becomes much easier as the number of models increases.
[0158] This paper proposes various dataset governance methods, providing corresponding solutions for issues such as missing data, outliers exceeding valid values, and duplicate data. The governed data can then be used normally in model training and inference. Furthermore, it offers a complete dataset governance pipeline, enabling users to easily address most data issues and reliably fill in missing data while effectively managing outliers.
[0159] Two training result models are saved. Users can choose the training result model with the smallest loss value to use directly in the next inference process, or they can compare the saved model parameter configuration dictionary to continue to optimize and train a better result model.
[0160] See Figure 5 , Figure 5 This is a schematic diagram of the framework of a model training device provided in some embodiments of this application.
[0161] The model training device 50 includes a data processing module 51, a sample determination module 52, and an execution module 53. The data processing module 51 acquires the user-configured target data processing method and processes the target dataset according to this method. The target data processing method includes several combined processing methods and / or several single processing methods. Combined processing methods include sequentially executing multiple single processing methods. Single processing methods include at least one of data merging, outlier handling, null value imputation, resampling, and data completion. The sample determination module 52 uses the processed target dataset as the sample dataset for training instances corresponding to the target model. The execution module 53 executes the training instances to train the target model using the target dataset and obtain training results.
[0162] Please see Figure 6 , Figure 6 This is a schematic diagram of the framework of an electronic device provided in some embodiments of this application.
[0163] Electronic device 60 includes memory 61 and processor 62. Processor 62 is used to execute program instructions stored in memory 61 to implement any of the above-mentioned model training methods. In a specific implementation scenario, electronic device 60 may include, but is not limited to: computer equipment, electrical equipment, microcomputer, desktop computer, server. In addition, electronic device 60 may also include mobile devices such as laptops and tablets, which are not limited here.
[0164] Specifically, processor 62 controls itself and memory 61 to implement any of the aforementioned model training methods. Processor 62 can also be referred to as a CPU (Central Processing Unit). Processor 62 may be an integrated circuit chip with signal processing capabilities. Processor 62 can also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. A general-purpose processor can be a microprocessor or any conventional processor. Furthermore, processor 62 can be implemented using integrated circuit chips.
[0165] Please see Figure 7 , Figure 7 This is a schematic diagram of the framework of a computer-readable storage medium provided in some embodiments of this application.
[0166] The computer-readable storage medium 70 stores program instructions 71 that can be executed by a processor. When the program instructions 71 are executed by the processor, they are used to implement any of the above-described model training methods.
[0167] The description of the various embodiments above tends to emphasize the differences between the various embodiments. The similarities or similarities between them can be referred to, and for the sake of brevity, they will not be repeated here.
[0168] In the several embodiments provided in this application, it should be understood that the disclosed methods and apparatus can be implemented in other ways. For example, the apparatus implementations described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, units or components may be combined or integrated into another subsystem, or some features may be ignored or not executed. Furthermore, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection of devices or units may be electrical, mechanical, or other forms.
[0169] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.
Claims
1. A model training method, characterized in that, The method includes: Obtain the target data processing method configured by the user, and process the target dataset according to the target data processing method. The target data processing method includes several combined processing methods and / or several single processing methods. The combined processing method includes sequentially executing the processing corresponding to multiple single processing methods. The single processing method includes at least one of data merging, outlier handling, null value imputation, resampling, and data completion. The target dataset after data processing is used as the sample dataset for training instances corresponding to the target model. The training instance is executed to train the target model using the target dataset and obtain the training results.
2. The method of claim 1, wherein, The step of obtaining the target data processing method configured by the user and processing the target dataset according to the target data processing method includes: Display a combined processing method selection page, which displays at least one preset combined processing method for the user to select; In response to the user's combination selection operation, select the plurality of combination processing methods from the at least one preset combination processing methods, and perform the processing corresponding to the plurality of combination processing methods on the target dataset; In response to the end of the combined selection operation, a single processing method selection page is displayed, in which at least one preset single processing method is displayed for the user to select. In response to a single selection operation, the plurality of single processing methods are selected from the at least one preset single processing method, and the processing corresponding to the plurality of single processing methods is performed on the target dataset.
3. The method according to any one of claims 1 to 2, characterized in that, The target dataset includes multiple data entries, each containing multiple fields; the multiple data entries are sorted by time. The data merging includes: merging multiple pieces of data to be merged into one integrated piece of data, wherein the multiple pieces of data to be merged have the same time; the value of each field in the integrated data is determined based on the value of the corresponding field in the multiple pieces of data to be merged; The outlier handling includes: selecting an outlier judgment field from the plurality of fields based on the processing configuration information corresponding to the target dataset; and replacing the values of the outlier judgment field that do not meet the preset field value requirements with empty values. The null value filling includes: taking the field with a null value as the field to be filled, determining the filling value corresponding to the field to be filled from the multiple data based on the processing configuration information corresponding to the target dataset, and filling the field to be filled with the filling value; The resampling includes: determining a preset sampling frequency based on the processing configuration information corresponding to the target dataset; and resampling the multiple data points according to the preset sampling frequency to obtain multiple resampled data points. The data completion includes: determining a preset time interval based on the processing configuration information corresponding to the target dataset; dividing the multiple data into several data groups according to the preset time interval; taking the data groups with data gaps as target data groups in sequence, and filling the target data groups with data from the previous data group.
4. The method according to any one of claims 1 to 3, characterized in that, Before using the target dataset after data processing as the sample dataset for training instances corresponding to the target model, the method further includes: In response to a model selection operation, a target model is determined from at least one preset model, and the current parameter configuration information of the target model is determined; and in response to a function selection operation, a target loss function is determined from at least one preset loss function. The training instance is constructed using the target model, the current parameter configuration information, and the target loss function. When the training instance is executed, the target model is trained based on the target loss function.
5. The method according to claim 4, characterized in that, The step of constructing the training instance using the target model, the current parameter configuration information, and the target loss function includes: The target model, the current parameter configuration information, and the target loss function are used as attribute values corresponding to the wrapper class; The class constructor method of the wrapper class is executed to construct the training instance.
6. The method of claim 4, wherein, The determination of the current parameter configuration information of the target model includes: Obtain the pre-stored parameter configuration information of the target model; and, in response to the user's parameter input operation, obtain the temporary parameter configuration information input by the user; Based on the temporary parameter configuration information and the pre-stored parameter configuration information, the current parameter configuration information of the target model is obtained.
7. The method according to claim 4, characterized in that, The method further includes: In response to a training resource creation command, the system obtains training resource attributes input by the user, obtains a training resource file based on the training resource attributes, and stores the training resource attributes and the training resource file. The training resource attributes and the training resource file are used to define the preset training resource. The preset training resources include at least one of the preset model, the preset loss function, and the preset dataset, wherein the preset dataset is provided for the user to select as the target dataset.
8. The method according to claim 7, characterized in that, The training resource attributes include the original storage path, and obtaining the training resource file based on the training resource attributes includes: obtaining the training resource file from the original storage path; And / or, storing the training resource attributes and the training resource file includes: In response to the training resource attribute including the resource storage method, a custom storage interface is constructed based on the resource storage method, and the custom storage interface is called to store the training resource attribute and the training resource file; In response to the fact that the training resource attributes do not include a resource storage method, the target storage address is obtained from the training resource attributes, and the training resource attributes and the training resource file are stored at the target storage address.
9. The method according to claim 8, characterized in that, The training resource attributes include a unique identifier attribute; the method further includes: In response to the training resource update command, obtain the update attributes and unique identifier attributes input by the user; In response to the existence of a preset training resource corresponding to the unique identifier attribute of the input, the currently stored attribute corresponding to the training resource is updated using the update attribute; If the preset training resource does not exist and corresponds to the unique identifier attribute of the input, an update failure message is displayed.
10. The method according to any one of claims 1 to 9, characterized in that, The training results include the first model obtained from the final training and the second model with the minimum loss; the method further includes: In response to the user's inference model selection operation, an inference model is selected from the first model and the second model; the inference model is used to perform parameter prediction based on the data to be processed to obtain the prediction result.
11. The method according to any one of claims 1 to 10, characterized in that, The aforementioned combined processing methods include a target combined processing method; the target combined processing method includes sequentially performing data merging, outlier handling, null value imputation, resampling, null value imputation, data completion, and null value imputation.
12. A model training apparatus, comprising: include: The data processing module is used to obtain the target data processing method configured by the user, and to process the target dataset according to the target data processing method. The target data processing method includes several combined processing methods and / or several single processing methods. The combined processing method includes sequentially executing the processing corresponding to multiple single processing methods. The single processing method includes at least one of data merging, outlier handling, null value imputation, resampling, and data completion. The sample determination module is used to use the target dataset after data processing as the sample dataset for training instances corresponding to the target model. The execution module is used to execute the training instance to train the target model using the target dataset and obtain the training result.
13. An electronic device, comprising: The method includes a memory and a processor, wherein the memory stores program instructions that, when executed by the processor, are used to perform the method described in any one of claims 1 to 11.
14. A computer-readable storage medium having stored thereon program instructions, wherein, When the program instructions are executed by the processor, they implement the method described in any one of claims 1 to 11.