Model training method and device, equipment and storage medium
By obtaining the data distribution information of single source data for preprocessing, the problems of poor data quality and messy distribution in large model training are solved, and efficient model training is achieved.
Patent Information
- Application Number
- CN202310988755.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-08-07
- Publication Date
- 2025-02-14
AI Technical Summary
In the prior art, due to the poor quality of massive data and the messy data distribution structure during training, the training takes longer and low efficiency.
By obtaining the data distribution information of single source data, preprocessing includes data filtering, data clearing and data merging, a high-quality model training sample set is constructed, and the target model is trained based on this sample set.
It effectively improves the quality and reasonable distribution of training data, shortens the model training time, and improves training efficiency.
Smart Images
Figure CN119441858A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of data processing technology, and in particular to a model training method, device, equipment and storage medium. Background Art
[0002] Big models are playing an increasingly important role in the Internet field, and the advantages of this technology will gradually be brought into play in terms of imagination, creativity and work costs. When training big models, it is necessary to obtain massive data to build training sets for training big models. However, the data sources of massive data are relatively complex. Massive data contains data from multiple sources, so the quality of massive data is low and the data distribution structure is messy. Therefore, the current training of big models takes a long time and the model training efficiency is low.
[0003] The above contents are only used to assist in understanding the technical solution of the present invention and do not constitute an admission that the above contents are prior art. Summary of the invention
[0004] The main purpose of the present invention is to provide a model training method, device, equipment and storage medium, aiming to solve the technical problems of poor quality of training data in the prior art, resulting in long model training time and low model training quality.
[0005] To achieve the above object, the present invention provides a model training method, which comprises the following steps:
[0006] Obtain data distribution information corresponding to at least one single source data;
[0007] Preprocessing the single-source data based on the data distribution information to obtain a model training sample set;
[0008] The target model is trained based on the model training sample set.
[0009] Optionally, the preprocessing includes at least one of the following: data filtering, data cleaning and data merging.
[0010] Optionally, preprocessing the single-source data based on the data distribution information to obtain a model training sample set includes:
[0011] Acquire a single-source model corresponding to each of the single-source data based on the data distribution information, wherein the single-source model is obtained by pre-training based on each of the single-source data;
[0012] Input each of the single-source data into the corresponding single-source model for data filtering;
[0013] A sample set is constructed based on the filtered single-source data to obtain a model training sample set.
[0014] Optionally, before acquiring the single-source model corresponding to each single-source data based on the data distribution information, the method includes:
[0015] Obtain the original model corresponding to each single source data;
[0016] The original model is trained based on each of the single-source data to obtain a single-source model corresponding to each of the single-source data.
[0017] Optionally, the obtaining of the original model corresponding to each single source data includes:
[0018] Obtain the training requirement information of the target model and the data feature information corresponding to each single source data;
[0019] Acquire a model architecture corresponding to each of the single-source data based on the training requirement information and the data feature information;
[0020] Based on the model architecture, an original model corresponding to each of the single-source data is obtained.
[0021] Optionally, inputting each of the single-source data into the corresponding single-source model for data filtering includes:
[0022] Input each of the single-source data into the corresponding single-source model for calculation to obtain the loss parameter and / or perplexity parameter of each of the single-source data;
[0023] The single-source data is filtered based on the loss parameter and / or the perplexity parameter.
[0024] Optionally, preprocessing the single-source data based on the data distribution information to obtain a model training sample set includes:
[0025] Acquire a loss parameter and / or a perplexity parameter of the single-source data based on the data distribution information;
[0026] Determining whether the loss parameter and / or the perplexity parameter exceeds a preset parameter threshold;
[0027] Determine the data to be cleared in the single-source data based on the judgment result;
[0028] Clearing the single-source data based on the data to be cleared;
[0029] A sample set is constructed based on the cleared single-source data to obtain a model training sample set.
[0030] Optionally, before determining whether the loss parameter and / or the perplexity parameter exceeds a preset parameter threshold, the step further includes:
[0031] Acquire a loss parameter and / or a perplexity parameter of the single-source data based on the data distribution information;
[0032] Acquire loss degree distribution information and / or perplexity distribution information of the single-source data based on the loss parameter and / or the perplexity parameter;
[0033] A parameter threshold is set based on the loss distribution information and / or the confusion distribution information.
[0034] Optionally, preprocessing the single-source data based on the data distribution information to obtain a model training sample set includes:
[0035] Performing data filtering and / or data clearing on each of the single-source data based on the data distribution information;
[0036] Merging the single-source data after data filtering and / or data cleaning;
[0037] A sample set is constructed based on the merged single-source data to obtain a model training sample set.
[0038] Optionally, before obtaining data distribution information corresponding to at least one single-source data, the method further includes:
[0039] Collecting data from at least one data source to obtain a data set corresponding to the data source;
[0040] The data set is preprocessed to obtain single-source data corresponding to the data source.
[0041] In addition, to achieve the above object, the present invention also proposes a model training device, the model training device comprising:
[0042] A data information acquisition module, used to acquire data distribution information corresponding to at least one single source data;
[0043] A data processing module, used to pre-process the single-source data based on the data distribution information to obtain a model training sample set;
[0044] The model training module is used to train the target model based on the model training sample set.
[0045] Optionally, the preprocessing includes at least one of the following: data filtering, data cleaning and data merging.
[0046] Optionally, the data processing module is also used to obtain a single source model corresponding to each of the single source data based on the data distribution information, wherein the single source model is obtained in advance based on training of each of the single source data; input each of the single source data into the corresponding single source model for data filtering; and construct a sample set based on the filtered single source data to obtain a model training sample set.
[0047] Optionally, the data processing module is further used to obtain an original model corresponding to each single source data; and to train the original model based on each single source data to obtain a single source model corresponding to each single source data.
[0048] Optionally, the data processing module is also used to obtain training requirement information of the target model and data feature information corresponding to each single source data; obtain the model architecture corresponding to each single source data based on the training requirement information and the data feature information; and obtain the original model corresponding to each single source data based on the model architecture.
[0049] Optionally, the data processing module is also used to input each of the single-source data into the corresponding single-source model for calculation to obtain the loss parameter and / or confusion parameter of each of the single-source data; and filter the single-source data based on the loss parameter and / or the confusion parameter.
[0050] Optionally, the data processing module is also used to obtain the loss parameter and / or confusion parameter of the single source data based on the data distribution information; determine whether the loss parameter and / or the confusion parameter exceeds a preset parameter threshold; determine the data to be cleared in the single source data based on the judgment result; clear the single source data based on the data to be cleared; construct a sample set based on the cleared single source data to obtain a model training sample set.
[0051] Optionally, the data processing module is also used to obtain the loss parameter and / or confusion parameter of the single source data based on the data distribution information; obtain the loss degree distribution information and / or confusion degree distribution information of the single source data based on the loss parameter and / or the confusion parameter; and set a parameter threshold based on the loss degree distribution information and / or the confusion distribution information.
[0052] In addition, to achieve the above-mentioned purpose, the present invention also proposes a model training device, which includes: a memory, a processor, and a model training program stored in the memory and executable on the processor, and the model training program is configured to implement the steps of the model training method described above.
[0053] In addition, to achieve the above-mentioned purpose, the present invention also proposes a storage medium, on which a model training program is stored, and when the model training program is executed by a processor, the steps of the model training method described above are implemented.
[0054] The present invention obtains data distribution information corresponding to at least one single-source data, pre-processes the single-source data based on the data distribution information, obtains a model training sample set, and trains a target model based on the model training sample set; because the present invention pre-processes the single-source data based on the data distribution information of the single-source data to obtain a model training sample set, and trains the target model based on the model training sample set, the data quality of the training data is effectively improved, the reasonable distribution of the data in the training set is ensured, and the model training efficiency is effectively improved. BRIEF DESCRIPTION OF THE DRAWINGS
[0055] Figure 1 It is a structural diagram of a model training device in a hardware operating environment involved in an embodiment of the present invention;
[0056] Figure 2 This is a flow chart of the first embodiment of the model training method of the present invention;
[0057] Figure 3 A schematic diagram of a single-source data processing flow chart of a first embodiment of a model training method of the present invention;
[0058] Figure 4 This is a flow chart of the second embodiment of the model training method of the present invention;
[0059] Figure 5 This is a flow chart of a third embodiment of the model training method of the present invention;
[0060] Figure 6 This is a structural block diagram of the first embodiment of the model training device of the present invention.
[0061] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION
[0062] It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.
[0063] Reference Figure 1 , Figure 1 A schematic diagram of the structure of a model training device in a hardware operating environment according to an embodiment of the present invention.
[0064] like Figure 1As shown, the model training device may include: a processor 1001, such as a central processing unit (CPU), a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. Among them, the communication bus 1002 is used to realize the connection and communication between these components. The user interface 1003 may include a display screen (Display), an input unit such as a keyboard (Keyboard), and the optional user interface 1003 may also include a standard wired interface and a wireless interface. The network interface 1004 may optionally include a standard wired interface and a wireless interface (such as a wireless fidelity (Wireless-Fidelity, Wi-Fi) interface). The memory 1005 may be a high-speed random access memory (Random Access Memory, RAM), or a stable non-volatile memory (Non-Volatile Memory, NVM), such as a disk storage. The memory 1005 may also be a storage device independent of the aforementioned processor 1001.
[0065] Those skilled in the art will understand that Figure 1 The structure shown in does not constitute a limitation on the model training device, and may include more or fewer components than shown in the figure, or a combination of certain components, or a different arrangement of components.
[0066] like Figure 1 As shown, the memory 1005 as a storage medium may include an operating system, a network communication module, a user interface module, and a model training program.
[0067] exist Figure 1 In the model training device shown, the network interface 1004 is mainly used for data communication with the network server; the user interface 1003 is mainly used for data interaction with the user; the processor 1001 and the memory 1005 in the model training device of the present invention can be set in the model training device, and the model training device calls the model training program stored in the memory 1005 through the processor 1001, and executes the model training method provided by the embodiment of the present invention.
[0068] The embodiment of the present invention provides a model training method, referring to Figure 2 , Figure 2 The figure is a flow chart of a first embodiment of a model training method of the present invention.
[0069] In this embodiment, the model training method includes the following steps:
[0070] Step S10: Obtain data distribution information corresponding to at least one single-source data.
[0071] It should be understood that the executor of the method of this embodiment can be a model training device with data processing, network communication and program running functions, such as a computer, or other devices or equipment that can achieve the same or similar functions. The above-mentioned model training device is used as an example for illustration here.
[0072] It should be noted that single-source data can be data from a single source, or data obtained from a single server or a single node. For example, single-source data may include random web pages, libraries, encyclopedias, codes, news sites, question-and-answer sites, etc., and may also involve multiple languages. The above-mentioned data distribution information may include information such as the data structure, data trend, data dispersion, and data distribution shape of the single-source data.
[0073] It should be understood that training a large model requires massive data, which includes single-source data from multiple data sources. This embodiment obtains data distribution information of each single-source data in the massive data, and pre-processes each single-source data based on the data distribution information, thereby ensuring the reasonable distribution of each single-source data, improving the data quality of the training data, and thus improving the efficiency of model training.
[0074] Furthermore, in order to effectively collect data, before the above step S10, the following may be included:
[0075] Collecting data from at least one data source to obtain a data set corresponding to the data source;
[0076] The data set is preprocessed to obtain single-source data corresponding to the data source.
[0077] It can be understood that the model training device obtains the training requirement information of the target model, determines the data requirement based on the training requirement information, determines the data source to be collected based on the data requirement, collects data from the data source, obtains the corresponding data set, pre-processes the data set to improve the data quality, removes invalid data (such as garbled data and dirty data) in the data set, and thus obtains the single-source data corresponding to the data source.
[0078] Step S20: pre-processing the single-source data based on the data distribution information to obtain a model training sample set.
[0079] It should be noted that the preprocessing may include data filtering, data cleaning, data merging, etc. The above-mentioned model training sample set may be a data set for training a target model, and the model training sample set includes the single-source data after preprocessing.
[0080] It should be understood that this embodiment pre-processes the single-source data based on data distribution information to obtain higher quality single-source data, merges the processed single-source data, and constructs a model training sample set, thereby ensuring the reasonable distribution of training data and improving the quality of training data.
[0081] It is understandable that, referring to Figure 3 , Figure 3 This is a schematic diagram of the single-source data processing flow. The model training device can pre-process the single-source data through the single-source model (for example, perform data filtering and data cleaning) to obtain new processed single-source data, update the single-source data to be trained based on the new single-source data, and then train the target model (large model) based on the updated single-source data. The single-source data is cyclically updated and processed based on the above process, thereby improving the data quality of the training sample set and improving the training efficiency of the large model.
[0082] Step S30: training the target model based on the model training sample set.
[0083] It should be noted that the target model can be a large model that needs to be trained. When training a large model, high-quality pre-training data plays a vital role in the training of the large model and determines the performance and efficiency of the model training.
[0084] It can be understood that this embodiment improves data quality by preprocessing single-source data, constructs a model training sample set with reasonable data distribution and higher quality, trains the target model based on the model training sample set, and improves the performance and effect of the model through continuous optimization and evaluation.
[0085] This embodiment obtains data distribution information corresponding to at least one single-source data, pre-processes the single-source data based on the data distribution information, obtains a model training sample set, and trains a target model based on the model training sample set; since the present invention pre-processes the single-source data based on the data distribution information of the single-source data to obtain a model training sample set, and trains the target model based on the model training sample set, the data quality of the training data is effectively improved, the reasonable distribution of the data in the training set is ensured, and the model training efficiency is effectively improved.
[0086] refer to Figure 4 , Figure 4 This is a flow chart of a second embodiment of a model training method of the present invention.
[0087] Based on the above first embodiment, in this embodiment, the preprocessing includes at least one of the following: data filtering, data clearing and data merging, and the step S20 includes:
[0088] Step S21: acquiring a single-source model corresponding to each of the single-source data based on the data distribution information, wherein the single-source model is obtained by pre-training based on each of the single-source data.
[0089] It should be noted that the single-source model can be a small single-source model pre-trained based on single-source data.
[0090] It can be understood that in order to effectively filter single-source data, the model training device obtains the original model of each single-source data, trains the corresponding original model based on the single-source data, until the original model converges, and obtains the single-source model.
[0091] Furthermore, in order to improve the data filtering efficiency, before the above step S21, it also includes:
[0092] Step S211: obtaining the original model corresponding to each single source data;
[0093] Step S212: training the original model based on each of the single-source data to obtain a single-source model corresponding to each of the single-source data.
[0094] It can be understood that this embodiment obtains task requirement information, determines the task type and task objectives based on the task requirement information, and thus determines the original model to be used based on the task type and task objectives, obtains the data source information and data type information of each single source data, and determines the original model corresponding to each single source data based on the data source information and data type information.
[0095] It should be understood that the convergence of the single-source model is mainly based on the loss value of the data. Using the gradient descent method, as the number of training steps increases, the loss value of the single-source data gradually stabilizes, indicating that the single-source model has converged on this data.
[0096] Furthermore, in order to accurately assign a suitable model to filter the data, the above step S211 may include:
[0097] Step S2111: obtaining training requirement information of the target model and data feature information corresponding to each single source data;
[0098] Step S2112: acquiring a model architecture corresponding to each of the single-source data based on the training requirement information and the data feature information;
[0099] Step S2113: obtaining the original model corresponding to each of the single-source data based on the model architecture.
[0100] It should be noted that the training requirement information may be requirement information such as data quality requirement, data perplexity requirement and data loss requirement, for example, requiring the perplexity and / or loss index of the single-source data to be lower than a threshold, etc. The above-mentioned data feature information may be feature information such as data distribution feature, data loss feature and data perplexity feature of the single-source data.
[0101] It should be understood that the model training device obtains the training requirement information of the target model to be trained (for example, the training requirement information may include task type, training indicators, performance requirements, etc.), performs data analysis on the single-source data, thereby extracting the data feature information of each single-source data, understanding the distribution and quality characteristics of the single-source data, and selecting a suitable single-source model architecture according to the training requirement information of the target model and the feature information of the single-source data. For example, the type of model, number of layers, number of neurons, etc. may be considered, and then a suitable original model is selected based on the single-source model architecture.
[0102] Step S22: input each of the single-source data into the corresponding single-source model for data filtering.
[0103] It is understandable that the trained single-source model is used to filter new single-source data. The single-source data to be filtered is input into the corresponding single-source model and classified or labeled according to the output of the single-source model. Through the prediction results of the single-source model, data that may be noise, abnormal, or does not meet specific conditions can be identified and filtered out.
[0104] In a specific implementation, the model training device filters the single-source data by utilizing a small single-source model that has converged during training, and calculates the perplexity and / or loss value of the single-source data, thereby filtering the single-source data.
[0105] Furthermore, in order to effectively improve data quality, the above step S22 may include:
[0106] Step S221: inputting each of the single-source data into the corresponding single-source model for calculation to obtain the loss parameter and / or perplexity parameter of each of the single-source data;
[0107] Step S222: filtering the single-source data based on the loss parameter and / or the perplexity parameter.
[0108] It should be noted that the loss parameter (Loss) of single-source data can be the error between the model prediction structure and the true label during model training, and the perplexity parameter (PPL) is an indicator for evaluating model performance.
[0109] It should be understood that this embodiment trains a converged small single-source model, filters the single-source data based on the single-source model, calculates the confusion parameters and / or average loss parameters of each single-source data, and then filters the single-source data based on the confusion parameters and / or average loss parameters, thereby ensuring the reasonable distribution of the single-source data and improving the data quality, thereby improving the pre-training effect of the large model.
[0110] Step S23: constructing a sample set based on the filtered single-source data to obtain a model training sample set.
[0111] It is understandable that, referring to Figure 3 , Figure 3 The figure is a schematic diagram of the data processing flow. The model training device can pre-process the single-source data through the single-source model (for example, perform data filtering and data cleaning) to obtain new processed single-source data, update the single-source data to be trained based on the new single-source data, and then train the target model (large model) based on the updated single-source data. The single-source data is cyclically updated and processed based on the above process, thereby improving the data quality of the training sample set and improving the training efficiency of the large model.
[0112] Furthermore, in order to improve the model training efficiency, the above step S20 may include:
[0113] Performing data filtering and / or data clearing on each of the single-source data based on the data distribution information;
[0114] Merging the single-source data after data filtering and / or data cleaning;
[0115] A sample set is constructed based on the merged single-source data to obtain a model training sample set.
[0116] It should be understood that the model training device obtains multiple single-source data, pre-processes each single-source data separately, obtains processed single-source data, merges the processed single-source data, constructs a model training sample set, and trains the target model based on the model training sample set.
[0117] In this embodiment, the single-source model corresponding to each single-source data is obtained based on the data distribution information, and the single-source model is obtained in advance based on the training of each single-source data. Each single-source data is input into the corresponding single-source model for data filtering, and a sample set is constructed based on the filtered single-source data to obtain a model training sample set. Since in this embodiment, each single-source data is input into the corresponding single-source model for data filtering, and the single-source data is filtered by using the single-source model, and a sample set is constructed based on the filtered single-source data to obtain a model training sample set, the reasonable distribution of training data is ensured, the quality of training data is improved, and the efficiency of model training is improved.
[0118] refer to Figure 5 , Figure 5 It is a flowchart diagram of the third embodiment of a model training method of the present invention.
[0119] Based on the above second embodiment, in this embodiment, step S20 may include:
[0120] Step S201: Obtain loss parameters and / or perplexity parameters of the single-source data based on the data distribution information.
[0121] It should be noted that the loss parameter (Loss) of single-source data can be the error between the model prediction structure and the true label during model training, and the perplexity parameter (PPL) is an indicator for evaluating model performance.
[0122] It should be understood that this embodiment trains a converged small single-source model, filters the single-source data based on the single-source model, calculates the confusion parameters and / or average loss parameters of each single-source data, and then cleans the single-source data based on the confusion parameters and / or average loss parameters to remove dirty data in the single-source data, obtain higher quality single-source data, ensure the reasonable distribution of single-source data, and thus improve the pre-training effect of the large model.
[0123] Step S202: Determine whether the loss parameter and / or the perplexity parameter exceeds a preset parameter threshold.
[0124] It should be noted that the parameter threshold may be a parameter threshold set in advance based on data distribution information of single-source data through random sampling, and the parameter threshold may include a loss parameter threshold and a perplexity parameter threshold.
[0125] Furthermore, in order to accurately set the parameter threshold, before the above step S202, the following steps may be included:
[0126] S2021: Obtaining a loss parameter and / or a perplexity parameter of the single-source data based on the data distribution information;
[0127] S2022: Obtain loss degree distribution information and / or confusion degree distribution information of the single-source data based on the loss parameter and / or the confusion parameter;
[0128] S2023: Setting a parameter threshold based on the loss distribution information and / or the confusion distribution information.
[0129] It should be noted that the loss distribution information may be the distribution information of the loss histogram of the single-source data constructed by random sampling, and the confusion distribution information may be the distribution information of the confusion histogram of the single-source data constructed by random sampling.
[0130] It should be understood that the present embodiment can obtain the loss distribution information and / or confusion distribution information of the single-source data by random sampling, for example, the loss histogram and / or confusion histogram of the single-source data can be obtained, and then transformed into a standard normal distribution. The extreme value parts with higher and lower scores after transformation (for example, 5% of the data before and after are respectively blocked) are filtered. In this part of the data, the low-scoring data may be dirty data, and the high-scoring data may be garbled data.
[0131] Step S203: determining the data to be cleared in the single-source data based on the judgment result.
[0132] It should be noted that the data to be cleared can be dirty data in which the loss parameter and / or confusion parameter in the single-source data exceeds the parameter threshold. The above dirty data can be data that exceeds the parameter threshold, such as data with a repeated sentence with a relatively low score, or garbled data (with a relatively high score). Since this type of data model is easier to learn patterns or is difficult to learn, it can be filtered out.
[0133] It should be understood that this embodiment filters the portion of the loss parameter and / or the confusion parameter in the single-source data that exceeds the parameter threshold to remove some dirty data in the single-source data, thereby obtaining higher quality single-source data.
[0134] Step S204: clearing the single-source data based on the data to be cleared.
[0135] It can be understood that this embodiment clears dirty data by pre-setting thresholds, clears and filters the extreme value parts in PPL and / or average Loss in the single-source data, and obtains higher quality single-source data, thereby ensuring the rationality of the data distribution of the training samples established subsequently.
[0136] Step S205: constructing a sample set based on the cleared single-source data to obtain a model training sample set.
[0137] It can be understood that the model training device cleans the single-source data to obtain higher quality single-source data, merges the cleaned single-source data, constructs a sample set, obtains a model training sample set, and trains a large model based on the model training sample set.
[0138] This embodiment obtains the loss parameter and / or confusion parameter of the single-source data based on the data distribution information, determines whether the loss parameter and / or the confusion parameter exceeds a preset parameter threshold, determines the data to be cleared in the single-source data based on the judgment result, clears the single-source data based on the data to be cleared, and constructs a sample set based on the cleared single-source data to obtain a model training sample set; because this embodiment obtains the loss parameter and / or confusion parameter of the single-source data, and then clears the single-source data based on the preset parameter threshold, thereby clearing the dirty data in the single-source data, and constructs a sample set based on the cleared single-source data to obtain a model training sample set, the cleanliness of the training data is effectively improved, the rationality of the data distribution is ensured, and the model training efficiency is improved.
[0139] In addition, an embodiment of the present invention further proposes a storage medium, on which a model training program is stored. When the model training program is executed by a processor, the steps of the model training method described above are implemented.
[0140] Since the storage medium adopts all the technical solutions of all the above embodiments, it has at least all the beneficial effects brought by the technical solutions of the above embodiments, which will not be described one by one here.
[0141] Reference Figure 6 , Figure 6 This is a structural block diagram of the first embodiment of the model training device of the present invention.
[0142] like Figure 6 As shown, the model training device proposed in the embodiment of the present invention includes:
[0143] A data information acquisition module 10 is used to acquire data distribution information corresponding to at least one single source data;
[0144] A data processing module 20, configured to pre-process the single-source data based on the data distribution information to obtain a model training sample set;
[0145] The model training module 30 is used to train the target model based on the model training sample set.
[0146] Furthermore, the preprocessing includes at least one of the following: data filtering, data cleaning and data merging.
[0147] Furthermore, the data processing module 20 is also used to obtain a single source model corresponding to each of the single source data based on the data distribution information, wherein the single source model is obtained in advance based on training of each of the single source data; input each of the single source data into the corresponding single source model for data filtering; and construct a sample set based on the filtered single source data to obtain a model training sample set.
[0148] Furthermore, the data processing module 20 is also used to obtain an original model corresponding to each single source data; and train the original model based on each single source data to obtain a single source model corresponding to each single source data.
[0149] Furthermore, the data processing module 20 is also used to obtain training requirement information of the target model and data feature information corresponding to each single source data; obtain the model architecture corresponding to each single source data based on the training requirement information and the data feature information; and obtain the original model corresponding to each single source data based on the model architecture.
[0150] Furthermore, the data processing module 20 is also used to input each of the single-source data into the corresponding single-source model for calculation to obtain the loss parameter and / or confusion parameter of each of the single-source data; and filter the single-source data based on the loss parameter and / or the confusion parameter.
[0151] Furthermore, the data processing module 20 is also used to obtain the loss parameter and / or confusion parameter of the single source data based on the data distribution information; determine whether the loss parameter and / or the confusion parameter exceeds a preset parameter threshold; determine the data to be cleared in the single source data based on the judgment result; clear the single source data based on the data to be cleared; and construct a sample set based on the cleared single source data to obtain a model training sample set.
[0152] Furthermore, the data processing module 20 is also used to obtain the loss parameter and / or confusion parameter of the single source data based on the data distribution information; obtain the loss degree distribution information and / or confusion degree distribution information of the single source data based on the loss parameter and / or the confusion parameter; and set a parameter threshold based on the loss degree distribution information and / or the confusion distribution information.
[0153] Furthermore, the data processing module 20 is also used to perform data filtering and / or data clearing on each of the single source data based on the data distribution information; to merge the single source data after data filtering and / or data clearing; and to construct a sample set based on the merged single source data to obtain a model training sample set.
[0154] Furthermore, the data information acquisition module 10 is used to collect data from at least one data source to obtain a data set corresponding to the data source; and pre-process the data set to obtain single-source data corresponding to the data source.
[0155] This embodiment obtains data distribution information corresponding to at least one single-source data, pre-processes the single-source data based on the data distribution information, obtains a model training sample set, and trains a target model based on the model training sample set; since the present invention pre-processes the single-source data based on the data distribution information of the single-source data to obtain a model training sample set, and trains the target model based on the model training sample set, the data quality of the training data is effectively improved, the reasonable distribution of the data in the training set is ensured, and the model training efficiency is effectively improved.
[0156] It should be understood that the above is only an example and does not constitute any limitation on the technical solution of the present invention. In specific applications, technicians in this field can make settings as needed, and the present invention does not limit this.
[0157] It should be noted that the workflow described above is merely illustrative and does not limit the scope of protection of the present invention. In practical applications, technicians in this field can select part or all of them according to actual needs to achieve the purpose of the present embodiment, and no limitation is made here.
[0158] In addition, for technical details not described in detail in this embodiment, reference can be made to the model training method provided in any embodiment of the present invention, and will not be repeated here.
[0159] The present invention also proposes A1, a model training method, the model training method comprising:
[0160] Obtain data distribution information corresponding to at least one single source data;
[0161] Preprocessing the single-source data based on the data distribution information to obtain a model training sample set;
[0162] The target model is trained based on the model training sample set.
[0163] A2. In the model training method described in A1, the preprocessing includes at least one of the following: data filtering, data cleaning, and data merging.
[0164] A3. The model training method as described in A2, wherein the single-source data is preprocessed based on the data distribution information to obtain a model training sample set, including:
[0165] Acquire a single-source model corresponding to each of the single-source data based on the data distribution information, wherein the single-source model is obtained by pre-training based on each of the single-source data;
[0166] Input each of the single-source data into the corresponding single-source model for data filtering;
[0167] A sample set is constructed based on the filtered single-source data to obtain a model training sample set.
[0168] A4. The model training method as described in A3, before obtaining the single-source model corresponding to each single-source data based on the data distribution information, comprises:
[0169] Obtain the original model corresponding to each single source data;
[0170] The original model is trained based on each of the single-source data to obtain a single-source model corresponding to each of the single-source data.
[0171] A5. The model training method as described in A4, wherein the original model corresponding to each single source data is obtained, comprising:
[0172] Obtain the training requirement information of the target model and the data feature information corresponding to each single source data;
[0173] Acquire a model architecture corresponding to each of the single-source data based on the training requirement information and the data feature information;
[0174] Based on the model architecture, an original model corresponding to each of the single-source data is obtained.
[0175] A6. The model training method as described in A3, wherein each of the single-source data is input into the corresponding single-source model for data filtering, comprising:
[0176] Input each of the single-source data into the corresponding single-source model for calculation to obtain the loss parameter and / or perplexity parameter of each of the single-source data;
[0177] The single-source data is filtered based on the loss parameter and / or the perplexity parameter.
[0178] A7. The model training method as described in any one of A2 to A6, wherein the single-source data is preprocessed based on the data distribution information to obtain a model training sample set, comprising:
[0179] Acquire a loss parameter and / or a perplexity parameter of the single-source data based on the data distribution information;
[0180] Determining whether the loss parameter and / or the perplexity parameter exceeds a preset parameter threshold;
[0181] Determine the data to be cleared in the single-source data based on the judgment result;
[0182] Clearing the single-source data based on the data to be cleared;
[0183] Construct a sample set based on the cleared single-source data to obtain a model training sample set
[0184] A8. The model training method as described in A7, before determining whether the loss parameter and / or the perplexity parameter exceeds a preset parameter threshold, further comprising:
[0185] Acquire a loss parameter and / or a perplexity parameter of the single-source data based on the data distribution information;
[0186] Acquire loss degree distribution information and / or perplexity distribution information of the single-source data based on the loss parameter and / or the perplexity parameter;
[0187] A parameter threshold is set based on the loss distribution information and / or the confusion distribution information.
[0188] A9. The model training method as described in A2, wherein the single-source data is preprocessed based on the data distribution information to obtain a model training sample set, including:
[0189] Performing data filtering and / or data clearing on each of the single-source data based on the data distribution information;
[0190] Merging the single-source data after data filtering and / or data cleaning;
[0191] A sample set is constructed based on the merged single-source data to obtain a model training sample set.
[0192] A10. The model training method as described in A1, before obtaining the data distribution information corresponding to at least one single-source data, further comprising:
[0193] Collecting data from at least one data source to obtain a data set corresponding to the data source;
[0194] The data set is preprocessed to obtain single-source data corresponding to the data source.
[0195] In addition, the present invention further proposes B11, a model training device, the model training device comprising:
[0196] A data information acquisition module, used to acquire data distribution information corresponding to at least one single source data;
[0197] A data processing module, used to pre-process the single-source data based on the data distribution information to obtain a model training sample set;
[0198] The model training module is used to train the target model based on the model training sample set.
[0199] B12. In the model training device as described in B11, the preprocessing includes at least one of the following: data filtering, data cleaning and data merging.
[0200] B13. In the model training device as described in B12, the data processing module is also used to obtain the single-source model corresponding to each of the single-source data based on the data distribution information, and the single-source model is obtained in advance based on the training of each of the single-source data; each of the single-source data is input into the corresponding single-source model for data filtering; and a sample set is constructed based on the filtered single-source data to obtain a model training sample set.
[0201] B14. In the model training device as described in B13, the data processing module is also used to obtain the original model corresponding to each single source data; the original model is trained based on each single source data to obtain the single source model corresponding to each single source data.
[0202] B15. In the model training device as described in B14, the data processing module is also used to obtain the training requirement information of the target model and the data feature information corresponding to each single source data; based on the training requirement information and the data feature information, the model architecture corresponding to each single source data is obtained; based on the model architecture, the original model corresponding to each single source data is obtained.
[0203] B16. In the model training device as described in B13, the data processing module is also used to input each of the single-source data into the corresponding single-source model for calculation to obtain the loss parameter and / or confusion parameter of each of the single-source data; and filter the single-source data based on the loss parameter and / or the confusion parameter.
[0204] B17. In the model training device as described in any one of B12 to 16, the data processing module is also used to obtain the loss parameter and / or confusion parameter of the single source data based on the data distribution information; determine whether the loss parameter and / or the confusion parameter exceeds a preset parameter threshold; determine the data to be cleared in the single source data based on the judgment result; clear the single source data based on the data to be cleared; and construct a sample set based on the cleared single source data to obtain a model training sample set.
[0205] B18. In the model training device as described in B17, the data processing module is also used to obtain the loss parameter and / or confusion parameter of the single source data based on the data distribution information; to obtain the loss degree distribution information and / or confusion degree distribution information of the single source data based on the loss parameter and / or the confusion parameter; and to set a parameter threshold based on the loss degree distribution information and / or the confusion distribution information.
[0206] The present invention also discloses C19, a model training device, which includes: a memory, a processor, and a model training program stored in the memory and executable on the processor, wherein the model training program is configured to implement the model training method described above.
[0207] The present invention also discloses D20, a storage medium, on which a model training program is stored, and when the model training program is executed by a processor, the model training method as described above is implemented.
[0208] In addition, it should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or system. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or system including the element.
[0209] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.
[0210] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as a read-only memory (ROM) / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present invention.
[0211] The above are only preferred embodiments of the present invention, and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.
Claims
1. A model training method, characterized in that: The model training method comprises: Obtain data distribution information corresponding to at least one single source data; Preprocessing the single-source data based on the data distribution information to obtain a model training sample set; The target model is trained based on the model training sample set.
2. The model training method according to claim 1, characterized in that: The preprocessing includes at least one of the following: data filtering, data cleaning and data merging.
3. The model training method according to claim 2, characterized in that: The preprocessing of the single-source data based on the data distribution information to obtain a model training sample set includes: Acquire a single-source model corresponding to each of the single-source data based on the data distribution information, wherein the single-source model is obtained by pre-training based on each of the single-source data; Input each of the single-source data into the corresponding single-source model for data filtering; A sample set is constructed based on the filtered single-source data to obtain a model training sample set.
4. The model training method according to claim 3, characterized in that: Before acquiring the single-source model corresponding to each of the single-source data based on the data distribution information, the method includes: Obtain the original model corresponding to each single source data; The original model is trained based on each of the single-source data to obtain a single-source model corresponding to each of the single-source data.
5. The model training method according to claim 4, characterized in that: The obtaining of the original model corresponding to each single source data includes Obtain the training requirement information of the target model and the data feature information corresponding to each single source data; Acquire a model architecture corresponding to each of the single-source data based on the training requirement information and the data feature information; Based on the model architecture, an original model corresponding to each of the single-source data is obtained.
6. The model training method according to claim 3, characterized in that: The step of inputting each of the single-source data into the corresponding single-source model for data filtering includes: Input each of the single-source data into the corresponding single-source model for calculation to obtain the loss parameter and / or perplexity parameter of each of the single-source data; The single-source data is filtered based on the loss parameter and / or the perplexity parameter.
7. The model training method according to any one of claims 2 to 6, characterized in that: The preprocessing of the single-source data based on the data distribution information to obtain a model training sample set includes: Acquire a loss parameter and / or a perplexity parameter of the single-source data based on the data distribution information; Determining whether the loss parameter and / or the perplexity parameter exceeds a preset parameter threshold; Determine the data to be cleared in the single-source data based on the judgment result; Clearing the single-source data based on the data to be cleared; A sample set is constructed based on the cleared single-source data to obtain a model training sample set.
8. A model training device, characterized in that: The model training device comprises: A data information acquisition module, used to acquire data distribution information corresponding to at least one single source data; A data processing module, used to pre-process the single-source data based on the data distribution information to obtain a model training sample set; The model training module is used to train the target model based on the model training sample set.
9. A model training device, characterized in that: The model training device includes: a memory, a processor, and a model training program stored in the memory and executable on the processor, wherein the model training program is configured to implement the model training method according to any one of claims 1 to 7.
10. A storage medium, characterized in that: The storage medium stores a model training program, and when the model training program is executed by the processor, the model training method according to any one of claims 1 to 7 is implemented.