Model training method and device, equipment and medium

Abstract

The invention discloses a model training method and device, equipment and a medium. The method comprises the following steps: acquiring at least one to-be-processed time slice, and screening from the at least one to-be-processed time slice to obtain a first time slice and a second time slice; downloading first data corresponding to the first time slice; in response to completion of downloading of the first data corresponding to the first time slice, training the data training model by adopting the first data corresponding to the first time slice, and downloading second data corresponding to the second time slice; and when the data training model is trained by the first data corresponding to the first time slice, updating the first time slice according to the second time slice. According to the embodiment of the invention, the model training efficiency can be improved.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a model training method, device, equipment and medium. Background Art

[0002] With the rapid development of science and technology, the types of data training models are gradually increasing. By importing a large amount of data into the data training model for training, the data training model can recognize the data features and generate data content corresponding to the data features.

[0003] Currently, the data corresponding to each time slice is downloaded and trained one by one according to the time sequence of each time slice.

[0004] However, the data of a time slice is downloaded and trained only after the data of a time slice is downloaded and trained, which leads to low efficiency of data training. Summary of the invention

[0005] The present invention provides a model training method, device, equipment and medium to improve the accuracy of model training.

[0006] In a first aspect, an embodiment of the present invention provides a model training method, the method comprising:

[0007] Obtain at least one time slice to be processed, and filter out a first time slice and a second time slice therefrom;

[0008] Downloading first data corresponding to a first time slice;

[0009] In response to the completion of downloading the first data corresponding to the first time slice, the data training model is trained using the first data corresponding to the first time slice, and the second data corresponding to the second time slice is downloaded at the same time;

[0010] When the training of the data training model for the first data corresponding to the first time slice is completed, the first time slice is updated according to the second time slice.

[0011] In a second aspect, an embodiment of the present invention further provides a model training device, the device comprising:

[0012] A data acquisition module, used to acquire at least one time slice to be processed, and filter out a first time slice and a second time slice therefrom;

[0013] A data download module, used for downloading first data corresponding to a first time slice;

[0014] A data training module, configured to, in response to the completion of downloading the first data corresponding to the first time slice, train the data training model using the first data corresponding to the first time slice, and download the second data corresponding to the second time slice at the same time;

[0015] The data updating module is used to update the first time slice according to the second time slice when the training of the data training model for the first data corresponding to the first time slice is completed.

[0016] In a third aspect, an embodiment of the present invention further provides a model training device, the model training device comprising:

[0017] at least one processor; and

[0018] a memory communicatively connected to at least one processor; wherein,

[0019] The memory stores a computer program that can be executed by at least one processor, and the computer program is executed by at least one processor so that the at least one processor can execute the model training method of any embodiment of the present invention.

[0020] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to implement the model training method of any embodiment of the present invention when executed by a processor.

[0021] The technical solution of the embodiment of the present invention obtains at least one time slice to be processed, and filters out the first time slice and the second time slice therefrom; downloads the first data corresponding to the first time slice; in response to the completion of downloading the first data corresponding to the first time slice, uses the first data corresponding to the first time slice to train the data training model, and downloads the second data corresponding to the second time slice at the same time; when the training of the data training model with the first data corresponding to the first time slice is completed, updates the first time slice according to the second time slice, so that the downloading data and training data steps corresponding to different time slices are executed in parallel, avoiding the situation where the data corresponding to one time slice is downloaded and trained before processing the data of the next time slice, thereby improving the efficiency of model training.

[0022] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0024] Figure 1is a flowchart of a model training method provided according to Embodiment 1 of the present invention;

[0025] Figure 2 is a flow chart of a model training method provided according to Embodiment 2 of the present invention;

[0026] Figure 3 is a structural diagram of a model training device provided according to an embodiment of the present invention;

[0027] Figure 4 It is a structural diagram of a model training device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0028] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0029] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0030] In the technical solution of the embodiment of the present invention, the acquisition, storage and application of the engine types involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0031] The information collected is information and data authorized by the user or fully authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure and application of relevant data comply with the relevant laws, regulations and standards of relevant countries and regions, take necessary confidentiality measures, do not violate public order and good customs, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0032] Embodiment 1

[0033] Figure 1This is a flow chart of a model training method provided in Embodiment 1 of the present invention. The embodiment of the present invention is applicable to the case of model training, and the method can be executed by a model training device, and the model training device can be implemented in the form of hardware and / or software.

[0034] See also Figure 1 The model training method shown includes:

[0035] S101. Obtain at least one time slice to be processed, and select a first time slice and a second time slice therefrom.

[0036] The time slice to be processed may be the time slice corresponding to the data to be trained on the model.

[0037] Specifically, a time slice is a microscopic period of CPU time allocated by a time-sharing operating system to each running process. When a process's time slice is used up, the operating system will suspend the process, save its current running state, and then allocate the CPU to another process to let it run for a time slice. A time slice can be divided into multiple time slices. The CPU time corresponding to each time slice may be different. At least one time slice can be obtained, and for each time slice, the time slice can be divided into at least one time slice, and the division method can be divided according to the length of different time periods. Obtain the preset filtering conditions, and in each time slice, filter out the first time slice and the second time slice according to the preset filtering conditions.

[0038] S102: Download first data corresponding to a first time slice.

[0039] The first data may be historical data corresponding to the first time slice.

[0040] Specifically, at least one historical data is obtained, and each historical data is divided according to the collection time of the historical data to obtain at least one historical data set. Different historical data sets correspond to different time slices. Different time slices correspond to at least one historical data, and there is a mapping relationship between the time slice and the corresponding historical data. The historical data corresponding to each time slice can be stored in the same public database, and when the data corresponding to the time slice is needed, it can be downloaded from the cloud database. The data corresponding to each time slice can be stored in the public database, by first downloading the data corresponding to each time slice to the local, and then performing data processing based on the local data. Therefore, according to the first time slice, a query is performed in the public database to obtain at least one historical data corresponding to the first time slice, determine each historical data as the first data, and download the first data corresponding to the first time slice to the local. The query method includes but is not limited to: a sequential search algorithm, a block search algorithm, a hash search algorithm, or a table query algorithm, etc., and the embodiment of the present invention does not limit this.

[0041] S103: In response to the completion of downloading the first data corresponding to the first time slice, the data training model is trained using the first data corresponding to the first time slice, and the second data corresponding to the second time slice is downloaded at the same time.

[0042] The data training model may be a model that identifies the features of the data corresponding to each time slice and is used for self-training. For example, the data training model includes, but is not limited to, a feedforward neural network model, a convolutional neural network model, a recurrent neural network model, a generative adversarial network model, or a pulse neural network model, etc., which is not limited in the embodiments of the present invention.

[0043] Specifically, when the first data corresponding to the first time slice is downloaded, the data training model corresponding to the first data corresponding to the first time slice is obtained. The first data corresponding to the first time slice is imported into the data training model, and the data training model is trained. When preparing to perform data training on the first data, the second data corresponding to the second time slice is downloaded. The two steps of downloading the second data and training the first data are performed in parallel. Since the types of resource requests for the two steps of training the first data and downloading the second data are different, in the data training process, the requested resources are used for calculation, and in the data download process, the requested resources are used for data communication. Therefore, the two steps can be executed in parallel, thereby improving the efficiency of data processing.

[0044] S104: When the training of the data training model for the first data corresponding to the first time slice is completed, the first time slice is updated according to the second time slice.

[0045] Specifically, when the first data corresponding to the first time slice completes the training of the data training model, the thread occupied by the first data training is idle. Therefore, the second time slice can be updated to the first time slice, and a time slice to be processed is reselected from the remaining time slices to be processed, and it is used as the second time slice. The idle thread corresponding to the original first time slice is used to process the data corresponding to the second time slice until the data corresponding to each time slice to be processed are trained. After the data corresponding to a time slice to be processed is processed, the time slice to be processed is deleted from each time slice to be processed, so as to avoid the time slice being repeatedly selected, reduce the amount of data to be processed, and improve the efficiency of data processing. The distributed data processing method is adopted, and data can be processed in parallel, avoiding the situation where the data of the same time slice corresponding to multiple processes are processed before the data processing of the next time slice is carried out, thereby reducing the waiting time of the system.

[0046] The technical solution of the embodiment of the present invention obtains at least one time slice to be processed, and filters out the first time slice and the second time slice therefrom; downloads the first data corresponding to the first time slice; in response to the completion of downloading the first data corresponding to the first time slice, uses the first data corresponding to the first time slice to train the data training model, and downloads the second data corresponding to the second time slice at the same time; when the training of the data training model with the first data corresponding to the first time slice is completed, updates the first time slice according to the second time slice, so that the downloading data and training data steps corresponding to different time slices are executed in parallel, avoiding the situation where the data corresponding to one time slice is downloaded and trained before processing the data of the next time slice, thereby improving the efficiency of model training.

[0047] Embodiment 2

[0048] Figure 2 This is a flow chart of a model training method provided in Embodiment 2 of the present invention. Based on the above embodiments, this embodiment of the present invention optimizes and improves the model training operation.

[0049] Further, the model training method is refined as "before downloading the first data corresponding to the first time slice, it also includes: obtaining the current access volume of the first data of the first time slice; determining the number of download threads and the number of training threads based on the current access volume; downloading the first data corresponding to the first time slice, including: using threads with the number of download threads to download the first data corresponding to the first time slice; using the first data corresponding to the first time slice to train the data training model, including: using threads with the number of training threads to train the data training model with the first data corresponding to the first time slice", so as to improve the operation of model training.

[0050] It should be noted that for the parts not described in detail in the embodiments of the present invention, reference may be made to the descriptions of other embodiments.

[0051] See also Figure 2 The model training method shown includes:

[0052] S201. Obtain at least one time slice to be processed, and select a first time slice and a second time slice therefrom.

[0053] S202: Obtain the current number of visits to the first data of the first time slice.

[0054] The current access amount may be the number of threads currently accessing the first data of the first time slice.

[0055] Specifically, different threads, different training machines or different processes may access the first data corresponding to the first time slice at the same time, and the number of accesses to the first data is relatively large; it is also possible that only a small number of different threads, different training machines or different processes access the first data at the same time. Different current access volumes correspond to different data processing steps. In order to refine the processing methods of data of different time slices, the current access volume of the first data of the first time slice is obtained. The acquisition method can be: querying data logs, querying system attribute information or detecting the number of current access program identifiers.

[0056] S203: Determine the number of download threads and the number of training threads according to the current access volume.

[0057] The number of download threads may be used to describe the number of threads for downloading the first data, and the number of training threads may be used to describe the number of threads for training the first data.

[0058] Specifically, if the current access volume is large, the efficiency of downloading the first data is low, indicating that even if the number of threads for downloading the first data is large, the download efficiency is difficult to improve; if the current access volume is small, the efficiency of downloading the first data is high, indicating that the number of threads for downloading the first data is large, and the download efficiency can also be improved. The total number of threads corresponding to each training machine remains unchanged. The number of threads corresponding to a training machine is the sum of the number of download threads and the number of training threads. When the number of download threads increases, the number of training threads decreases. If the number of training threads increases, the number of download threads decreases. When the current access volume is large, the download efficiency after the number of download threads is reduced by a few is within a small range compared to the download efficiency without reduction. In this case, the number of download threads can be reduced, and the reduced download threads can be used for data training, which can reduce the data training time and improve the efficiency of data processing. According to the current access volume, the number of download threads and the number of training threads are adaptively adjusted to ensure that the thread allocation is in line with the current system operation status and the real-time status of the system operation, thereby improving the accuracy of data processing.

[0059] S204: Use threads with the same number of download threads to download first data corresponding to the first time slice.

[0060] Specifically, according to the current access volume corresponding to the first time slice, the number of download threads and the number of training threads for downloading the first data are determined. According to the number of download threads, the number of download threads is selected from each thread corresponding to the training machine, and the first data corresponding to the first time slice is downloaded from the public database through each selected thread.

[0061] S205. In response to the completion of downloading the first data corresponding to the first time slice, use threads with the same number of training threads to train the data training model using the first data corresponding to the first time slice, and download the second data corresponding to the second time slice at the same time.

[0062] Specifically, a number of training threads are selected from each thread corresponding to the training machine. The thread used for training is different from the thread used for downloading, and one thread cannot be selected as a download thread and a training thread at the same time. After the download of the first data corresponding to the first time slice is completed, the threads with the number of training threads import the first data corresponding to the first time slice into the data training model to train the model. Since the download thread has completed downloading the first data, the download thread is idle, and the download thread is used to download the second data corresponding to the second time slice. After the training of the first data corresponding to the first time slice is completed, at this time, the second data has been downloaded, and the thread used to train the first data can be directly used to train the second data without waiting for the download of the second data, which saves thread waiting time and improves the efficiency of model training.

[0063] S206: When the training of the data training model for the first data corresponding to the first time slice is completed, the first time slice is updated according to the second time slice.

[0064] The embodiment of the present invention processes historical operation data to obtain the current access volume of the first data of the first time slice; determines the number of download threads and the number of training threads according to the current access volume, uses threads with the number of download threads to download the first data corresponding to the first time slice, uses threads with the number of training threads to train the data training model with the first data corresponding to the first time slice, allocates threads according to the real-time current access volume, executes different data processing methods for different real-time data, refines the model training steps, and improves the accuracy of model training.

[0065] Optionally, the data training model is trained using the first data corresponding to the first time slice, and the second data corresponding to the second time slice is downloaded at the same time, including: obtaining a target training thread corresponding to the number of training threads, and obtaining a first start time corresponding to the first time slice; starting from the first start time corresponding to the first time slice, training the data training model by calling the first data through the target training thread; obtaining the training start time; and downloading the second data corresponding to the second time slice according to the training start time.

[0066] The target training thread may be the number of training threads selected in the training machine. The first start time may be the time when the first data starts training the data training model. The training start time may be the time when the data training model starts training.

[0067] Specifically, obtain the target training thread corresponding to the number of training threads. Obtain a preset time, which is artificially preset. When the download of the first data is completed, obtain the download completion time of the first data, and determine the download completion time of the first data as the first start time. If the training thread is successfully called according to the first start time corresponding to the first time slice, the first data is called through the target training thread to train the data training model. If the training thread fails to be called, the first start time is updated according to the preset time, and the training thread is called again, and the first data is called through the target training thread to train the data training model; obtain the training start time, and according to the training start time, the thread that finishes downloading the first data is used to download the second data corresponding to the second time slice.

[0068] By obtaining the target training thread corresponding to the number of training threads and the first start time corresponding to the first time slice; starting from the first start time corresponding to the first time slice, the data training model is trained by calling the first data through the target training thread; the training start time is obtained; according to the training start time, the second data corresponding to the second time slice is downloaded, and the multiple time control processes are run steadily to avoid the data processing terminal and the data training failure.

[0069] Optionally, obtaining the first start time corresponding to the first time slice includes: obtaining the initial time point corresponding to the first time slice; obtaining the target download thread according to the number of download threads; starting from the initial time point, downloading the data corresponding to the first time slice through the target download thread to obtain the first data; obtaining the download completion time; and determining the download completion time as the first start time.

[0070] The initial time point may be a time point at which the first data is prepared to be downloaded. The target download thread may be a number of download threads selected from the training machine. The download completion time may be a time at which the first data is downloaded.

[0071] Specifically, the initial time point corresponding to the first time slice is obtained, which is used to control the starting processing time of the data corresponding to each time slice to be processed, so that different training machines have different starting processing times, which is used for data processing diversion, reducing the concurrency of data at the same time, and ensuring the safe operation of the training machine. According to the number of download threads, a target download thread is screened in a training machine, and starting from the initial time point, the data corresponding to the first time slice is downloaded through the target download thread to obtain the first data. The download completion time is obtained; the download completion time is determined as the first starting time. The data training type corresponding to the data training model is obtained. If the data type of the first data is different from the data training type corresponding to the data training model, the data type of the first data is converted into the data training type corresponding to the data training model, and the conversion completion time is updated to the first starting time.

[0072] By obtaining the initial time point corresponding to the first time slice; obtaining the target download thread according to the number of download threads; starting from the initial time point, downloading the data corresponding to the first time slice through the target download thread to obtain the first data; obtaining the download completion time; determining the download completion time as the first starting time, controlling the execution time of different training machines, and reducing the concurrent processes of processing data.

[0073] Optionally, the number of download threads and the number of training threads are determined based on the current access volume, including: obtaining an initial training thread and an initial download thread; obtaining an access volume threshold; comparing the current access volume with the access volume threshold to obtain a threshold comparison result; determining thread adjustment information based on the threshold comparison result; updating the initial training thread and the initial download thread based on the thread adjustment information to obtain a target training thread and a target download thread; and determining the number of download threads and the number of training threads based on the target training thread and the target download thread.

[0074] The initial training thread may be a preset thread for data training. The initial download thread may be a preset thread for data downloading. The access volume threshold may be a preset threshold of the current access volume. The threshold comparison result may be a result of comparing the current access volume with the access volume threshold. The thread adjustment information may be information for adjusting the thread allocation result.

[0075] Specifically, the initial training thread and the initial download thread are obtained. The access volume threshold is obtained, and the access volume threshold can be preset based on the experience of industry experts. The current access volume is compared with the access volume threshold to obtain the threshold comparison result. If the current access volume is large, it indicates that the number of download threads does not need to be too many, and the number of download threads can be reduced. If the current access volume is small, it indicates that more threads can be used to download data. According to the threshold comparison result, the thread adjustment information is determined. According to the thread adjustment information, the number of initial training threads and initial download threads is adjusted to obtain the target training thread and the target download thread; the target training thread and the target download thread are counted to determine the number of download threads and the number of training threads.

[0076] By obtaining the initial training thread and the initial download thread; obtaining the access volume threshold; comparing the current access volume with the access volume threshold to obtain the threshold comparison result; determining the thread adjustment information according to the threshold comparison result; updating the initial training thread and the initial download thread according to the thread adjustment information to obtain the target training thread and the target download thread; determining the number of download threads and the number of training threads for the target training thread and the target download thread, performing different operations through different threshold comparison results, refining the steps of model training and improving the accuracy of model training.

[0077] Optionally, thread adjustment information is determined based on the threshold comparison result, including: if the threshold comparison result is that the current access volume is greater than or equal to the access volume threshold, then the thread adjustment number is obtained, the thread adjustment number of threads is reduced in the initial download thread, and the reduced thread adjustment number of threads is added to the initial training thread; if the threshold comparison result is that the current access volume is less than the access volume threshold, then the initial training thread is used as the target training thread, and the initial download thread is used as the target download thread.

[0078] Specifically, if the threshold comparison result is that the current access volume is greater than or equal to the access volume threshold, the thread adjustment number is obtained. When the total number of threads of the training machine is less than or equal to 20, the thread adjustment number is 1. If the total number of threads of the training machine is greater than 20, the thread adjustment number is 2. The thread adjustment number is used to describe the number of threads reduced or increased each time. Reduce the thread adjustment number of threads in the initial download thread, and add the reduced thread adjustment number of threads to the initial training thread; if the threshold comparison result is that the current access volume is less than the access volume threshold, it indicates that data can be downloaded according to the preset number of download threads, and the initial training thread is used as the target training thread, and the initial download thread is used as the target download thread.

[0079] If the threshold comparison result is that the current access volume is greater than or equal to the access volume threshold, the number of adjusted threads is obtained, the number of threads adjusted is reduced in the initial download thread, and the reduced number of threads adjusted is added to the initial training thread; if the threshold comparison result is that the current access volume is less than the access volume threshold, the initial training thread is used as the target training thread, and the initial download thread is used as the target download thread. The number of different types of threads can be allocated according to real-time data, and the number of threads can be adjusted in time to make the threads more in line with current operating needs and improve the efficiency of model training.

[0080] Optionally, at least one time slice to be processed is obtained, and a first time slice and a second time slice are obtained by screening therefrom, including: obtaining at least one time slice to be processed; obtaining a preset processing order; sorting each time slice according to each time slice to be processed and the preset processing order to obtain a time series; clustering each time slice to be processed in the time series according to each time slice to be processed, and updating the time series; determining the first time clustering set in the updated time series as the first time slice, and determining the second time clustering set as the second time slice.

[0081] Specifically, at least one time slice to be processed is obtained. Each time slice corresponds to a slice identifier. A preset processing order is obtained, and the preset processing order is used to describe the order of slice identifiers corresponding to each time slice. The time slices are sorted according to each time slice to be processed and the preset processing order to obtain a time series. According to each time slice to be processed, the time slices to be processed in the time series are clustered, and the time slices with continuous positions with the same time slice identifier in the time series are determined as one category. When downloading data, the data of this category are downloaded at one time, and the sequence composed of the clustered time slices is updated to update the time series; the first time clustering set in the updated time series is determined as the first time slice, and the second time clustering set is determined as the second time slice. Different time slices correspond to different data, and data processing can be performed according to the priority of the required data type to be processed, so that data processing is more flexible.

[0082] By obtaining at least one time slice to be processed; obtaining a preset processing order; sorting each time slice according to each time slice to be processed and the preset processing order to obtain a time series; clustering each time slice to be processed in the time series according to each time slice to be processed, and updating the time series; determining the first time clustering set in the updated time series as the first time slice, determining the second time clustering set as the second time slice, and sorting each time slice, different data can be trained according to the preset execution order, thereby improving the accuracy of personalized training of the model.

[0083] Embodiment 3

[0084] Figure 3 This is a schematic diagram of the structure of a model training device provided in Embodiment 3 of the present invention. The embodiment of the present invention is applicable to the case of model training, the device can execute the model training method, and the device can be implemented in the form of hardware and / or software.

[0085] See also Figure 3 The model training device shown includes: a data acquisition module 301, a data download module 302, a data training module 303 and a data update module 304, wherein:

[0086] The data acquisition module 301 is used to acquire at least one time slice to be processed, and filter out a first time slice and a second time slice therefrom;

[0087] A data download module 302, used to download first data corresponding to a first time slice;

[0088] A data training module 303 is used to train a data training model using the first data corresponding to the first time slice in response to the completion of downloading the first data corresponding to the first time slice, and download the second data corresponding to the second time slice at the same time;

[0089] The data updating module 304 is used to update the first time slice according to the second time slice when the training of the data training model for the first data corresponding to the first time slice is completed.

[0090] The technical solution of the embodiment of the present invention obtains at least one time slice to be processed, and filters out the first time slice and the second time slice therefrom; downloads the first data corresponding to the first time slice; in response to the completion of downloading the first data corresponding to the first time slice, uses the first data corresponding to the first time slice to train the data training model, and downloads the second data corresponding to the second time slice at the same time; when the training of the data training model with the first data corresponding to the first time slice is completed, updates the first time slice according to the second time slice, so that the downloading data and training data steps corresponding to different time slices are executed in parallel, avoiding the situation where the data corresponding to one time slice is downloaded and trained before processing the data of the next time slice, thereby improving the efficiency of model training.

[0091] Optionally, the model training device also includes:

[0092] A pageview acquisition module: used to acquire the current pageview of the first data of the first time slice;

[0093] The quantity determination module is used to determine the number of download threads and the number of training threads according to the current access volume.

[0094] Optionally, the data download module 302 includes:

[0095] The thread downloading unit is used to download the first data corresponding to the first time slice by using threads having the same number of download threads.

[0096] Optionally, the data training module 303 includes:

[0097] The model training unit is used to train the data training model using threads with the same number of training threads and the first data corresponding to the first time slice.

[0098] Optionally, the data training module 303 includes:

[0099] A time acquisition unit, used to acquire a target training thread corresponding to the number of training threads, and to acquire a first start time corresponding to a first time slice;

[0100] A model calling unit, configured to call the first data through a target training thread to train the data training model starting from a first start time corresponding to the first time slice;

[0101] A start time determination unit, used to obtain the training start time;

[0102] The training data downloading unit is used to download the second data corresponding to the second time slice according to the training start time.

[0103] Optional, time acquisition unit, specifically used for:

[0104] Get the initial time point corresponding to the first time slice;

[0105] According to the number of download threads, obtain the target download thread;

[0106] Starting from the initial time point, the data corresponding to the first time slice is downloaded through the target download thread to obtain the first data;

[0107] Get the download completion time;

[0108] The download completion time is determined as the first start time.

[0109] Optional, quantity determination module, specifically for:

[0110] An initial thread acquisition unit, used to acquire an initial training thread and an initial download thread;

[0111] A threshold value obtaining unit, used for obtaining a visit volume threshold value;

[0112] A threshold comparison unit, used to compare the current page views with the page view threshold to obtain a threshold comparison result;

[0113] A thread adjustment unit, used to determine thread adjustment information according to a threshold comparison result;

[0114] An initial thread updating unit, used for updating an initial training thread and an initial downloading thread according to the thread adjustment information to obtain a target training thread and a target downloading thread;

[0115] The thread quantity determination unit is used to determine the number of download threads and the number of training threads from the target training thread and the target download thread.

[0116] Optional, thread adjustment unit, specifically used for:

[0117] If the threshold comparison result is that the current access volume is greater than or equal to the access volume threshold, the thread adjustment number is obtained, the thread adjustment number of threads is reduced in the initial download thread, and the reduced thread adjustment number of threads is added to the initial training thread;

[0118] If the threshold comparison result is that the current access volume is less than the access volume threshold, the initial training thread is used as the target training thread, and the initial download thread is used as the target download thread.

[0119] Optionally, the data acquisition module 301 is specifically used for:

[0120] Get at least one time slice to be processed;

[0121] Get the preset processing order;

[0122] Sort the time slices according to the time slices to be processed and the preset processing order to obtain a time series;

[0123] According to each time slice to be processed, clustering processing is performed on each time slice to be processed in the time series, and the time series is updated;

[0124] The first time cluster set in the updated time series is determined as the first time slice, and the second time cluster set is determined as the second time slice.

[0125] The model training device provided in the embodiment of the present invention can execute the model training method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects for executing the model training method.

[0126] Embodiment 4

[0127] Figure 4 A schematic diagram of the structure of a model training device 400 that can be used to implement an embodiment of the present invention is shown.

[0128] like Figure 4 As shown, the model training device 400 includes at least one processor 401, and a memory connected to the at least one processor 401, such as a read-only memory (ROM) 402, a random access memory (RAM) 403, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 401 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 402 or the computer program loaded from the storage unit 408 to the random access memory (RAM) 403. In RAM403, various programs and data required for the operation of the model training device 400 can also be stored. The processor 401, ROM402 and RAM403 are connected to each other via a bus 404. An input / output (I / O) interface 405 is also connected to the bus 404.

[0129] Multiple components in the model training device 400 are connected to the I / O interface 405, including: an input unit 406, such as a keyboard, a mouse, etc.; an output unit 407, such as various types of displays, speakers, etc.; a storage unit 408, such as a disk, an optical disk, etc.; and a communication unit 409, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 409 allows the model training device 400 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0130] Processor 401 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of processor 401 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. Processor 501 performs the various methods and processes described above, such as a model training method.

[0131] In some embodiments, the model training method may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 408. In some embodiments, part or all of the computer program may be loaded and / or installed on the model training device 400 via ROM 402 and / or communication unit 409. When the computer program is loaded into RAM 403 and executed by processor 401, one or more steps of the model training method described above may be performed. Alternatively, in other embodiments, processor 401 may be configured to perform the model training method in any other suitable manner (e.g., by means of firmware).

[0132] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0133] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0134] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in combination with an instruction execution system, device or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0135] To provide interaction with a user, the systems and techniques described herein may be implemented on a model training device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to a user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which a user can provide input to the model training device. Other types of devices may also be used to provide interaction with a user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0136] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0137] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship between the client and the server is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or a cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS (Virtual Private Server) services.

[0138] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0139] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A model training method, characterized in that: The method comprises: Obtain at least one time slice to be processed, and filter out a first time slice and a second time slice therefrom; Downloading first data corresponding to the first time slice; In response to the completion of downloading the first data corresponding to the first time slice, training the data training model using the first data corresponding to the first time slice, and downloading the second data corresponding to the second time slice; When the data training model is trained using the first data corresponding to the first time slice, the first time slice is updated according to the second time slice.

2. The method according to claim 1, characterized in that Before downloading the first data corresponding to the first time slice, the method further includes: Obtaining the current number of visits to the first data of the first time slice; Determine the number of download threads and the number of training threads according to the current access volume; The downloading the first data corresponding to the first time slice includes: Using the threads with the number of download threads to download the first data corresponding to the first time slice; The using the first data corresponding to the first time slice to train the data training model includes: The data training model is trained using the threads having the number of training threads and the first data corresponding to the first time slice.

3. The method according to claim 2, characterized in that The adopting the first data corresponding to the first time slice to train the data training model, and downloading the second data corresponding to the second time slice, includes: Obtaining target training threads corresponding to the number of training threads, and obtaining a first starting time corresponding to the first time slice; Starting from a first start time corresponding to the first time slice, calling the first data through the target training thread to train the data training model; Get the training start time; According to the training start time, second data corresponding to the second time slice is downloaded.

4. The method according to claim 3, characterized in that The obtaining the first start time corresponding to the first time slice includes: Obtaining an initial time point corresponding to the first time slice; According to the number of download threads, obtaining a target download thread; Starting from the initial time point, downloading the data corresponding to the first time slice through the target download thread to obtain first data; Get the download completion time; The download completion time is determined as the first start time.

5. The method according to claim 2, characterized in that: Determining the number of download threads and the number of training threads according to the current access volume includes: Get the initial training thread and the initial download thread; Get the visit volume threshold; Compare the current page views with the page view threshold to obtain a threshold comparison result; Determining thread adjustment information according to the threshold comparison result; The initial training thread and the initial download thread are updated according to the thread adjustment information to obtain a target training thread and a target download thread; The target training threads and the target download threads are used to determine the number of download threads and the number of training threads.

6. The method according to claim 5, characterized in that Determining thread adjustment information according to the threshold comparison result includes: If the threshold comparison result is that the current access volume is greater than or equal to the access volume threshold, obtaining the thread adjustment number, reducing the thread adjustment number of threads in the initial download thread, and adding the reduced thread adjustment number of threads to the initial training thread; If the threshold comparison result is that the current page views are less than the page view threshold, the initial training thread is used as the target training thread, and the initial download thread is used as the target download thread.

7. The method according to claim 1, characterized in that The obtaining of at least one time slice to be processed and screening out the first time slice and the second time slice therefrom includes: Get at least one time slice to be processed; Get the preset processing order; Sorting the time slices according to the to-be-processed time slices and the preset processing order to obtain a time series; According to each of the to-be-processed time slices, clustering processing is performed on each of the to-be-processed time slices in the time series, and updating the time series; The first time cluster set in the updated time series is determined as the first time slice, and the second time cluster set is determined as the second time slice.

8. A model training device, characterized in that: The device comprises: A data acquisition module, used to acquire at least one time slice to be processed, and filter out a first time slice and a second time slice therefrom; A data downloading module, used for downloading the first data corresponding to the first time slice; A data training module, configured to, in response to the completion of downloading the first data corresponding to the first time slice, train a data training model using the first data corresponding to the first time slice, and download the second data corresponding to the second time slice at the same time; A data updating module is used to update the first time slice according to the second time slice when the data training model is trained with the first data corresponding to the first time slice.

9. A model training device, characterized in that: The model training device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the model training method described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the model training method described in any one of claims 1 to 7 when executed.

Images