Data processing method and apparatus
By introducing a data processing engine to refine the sample data, the problem of insufficient support for diverse data sources in the model training framework is solved, and efficient model training and resource optimization are achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- BEIJING VOLCANO ENGINE TECH CO LTD
- Filing Date
- 2022-12-27
- Publication Date
- 2026-05-12
AI Technical Summary
Existing model training frameworks lack data mixing capabilities and cannot support flexible and diverse data sources, resulting in high development costs and training delays, as well as wasted computing and storage resources.
It introduces a feature-rich data processing engine to achieve refined sample data shuffling, mixing, and preprocessing, supports training from multiple data sources, and processes data mixing and model training in parallel.
It improves model training efficiency, reduces development costs, reduces training latency and resource waste, and enhances model training results.
Smart Images
Figure CN115879006B_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present disclosure relates to the technical field of computer, and particularly relates to a data processing method and device. BACKGROUND
[0002] During model training, the arrangement of sample data is one of important factors affecting model iteration. Therefore, in order to improve the model training effect, rich data sources are introduced, and the sample data in the data sources are mixed and preprocessed to obtain more flexible sample data to achieve better model training effect.
[0003] In the traditional way, data preprocessing is completed by a data preprocessing framework in the pre-processing stage, and then the preprocessed sample data is delivered to a model training framework, and model training is performed by a trainer in the model training framework. In the case that there are many data sources and the data types of sample data are rich, multiple types of data preprocessing frameworks need to be deployed to adapt to different data sources for mixing and preprocessing, which results in extremely high cost of supporting flexible and diverse data sources. SUMMARY
[0004] In order to solve the above technical problems and realize support for flexible and diverse data sources and improve the model training effect, the present disclosure provides a data processing method and device.
[0005] In a first aspect, the present disclosure provides a data processing method, comprising:
[0006] calling a scheduling management module in a data processing system to perform slicing processing on a sample data set corresponding to a model training task to obtain initial slice data; the initial slice data includes sample data of one or more data sources;
[0007] calling a data processing engine in the data processing system to perform mixing processing and / or preprocessing on the initial slice data to obtain target slice data meeting data requirements of a model training module; the data processing engine supports reading data of different data sources and performing mixing processing and preprocessing on data of different data sources; the mixing processing is used to indicate to disperse data order of sample data included in the initial slice data; the preprocessing is used to indicate to perform information conversion and / or sample filtering on the initial slice data;
[0008] performing model training by inputting the target slice data into the model training module to obtain a model meeting a training end condition.
[0009] In some embodiments, the calling the data processing engine in the data processing system to perform mixing and / or preprocessing on the initial slice data comprises:
[0010] The data processing engine is invoked to sort data in the one or more data sources by time, and sample data contained in the initial slice data is aggregated based on a sorting result;
[0011] For each initial slice data, the data processing engine is invoked to scatter sample data contained in the initial slice data by row / file to obtain mixed initial slice data.
[0012] In some embodiments, the invocation of the data processing engine in the data processing system to perform mixing processing and / or preprocessing on the initial slice data to obtain target slice data that meets the data requirements of the model training module includes:
[0013] The data processing engine is invoked to preprocess first data in the initial slice data to obtain third data that meets the data requirements of the model training module, and it is checked whether second data that does not need to be preprocessed is included in the initial slice data;
[0014] The data processing engine is invoked to merge the third data and the second data that does not need to be preprocessed obtained by checking to obtain the target slice data, and the second data meets the data requirements of the model training module.
[0015] In some embodiments, the invocation of the data processing engine by the scheduling management module to perform mixing processing and / or preprocessing on the initial slice data to obtain target slice data that meets the data requirements of the model training module includes:
[0016] Based on a preset concurrency number, a plurality of data processing engines in the data processing system are invoked in parallel to perform mixing processing and / or preprocessing on a plurality of initial slice data to obtain target slice data output by each data processing engine.
[0017] In some embodiments, it further includes:
[0018] The scheduling management module is invoked to check the execution state of the data processing engine;
[0019] When the execution state of the data processing engine is detected to be a failure state, a preset restart strategy is used to determine whether to restart the data processing engine to reprocess the corresponding initial slice data or end the mixing processing and / or preprocessing on the corresponding initial slice data.
[0020] In some embodiments, the determination of whether to restart the data processing engine to reprocess the corresponding initial slice data or end the mixing processing and / or preprocessing on the corresponding initial slice data based on the preset restart strategy includes:
[0021] Determine whether this failure has reached the maximum number of failures indicated by the preset restart policy;
[0022] If the maximum number of failures is reached, the data processing engine is restarted to reprocess and / or preprocess the corresponding initial slice data.
[0023] If the maximum number of failures is not reached, the mixing and / or preprocessing of the corresponding slice data is terminated, and the failure results are transmitted to the model training module.
[0024] In some embodiments, the step of inputting the target slice data into the model training module for model training includes:
[0025] The target slice data output by the multiple data processing engines are sequentially input into the data queue according to the order of slice processing;
[0026] The model training module is invoked to read the target slice data from the data queue for model training.
[0027] In some embodiments, the method further includes: during the training process, deleting the target slice data trained by the model training module from the data processing system.
[0028] Secondly, this disclosure provides a data processing apparatus, including:
[0029] The scheduling and management module is used to slice the sample dataset corresponding to the model training task to obtain initial slice data; the initial slice data includes sample data from one or more data sources.
[0030] A data processing engine is used to perform mixing and / or preprocessing on the initial slice data to obtain target slice data that meets the data requirements of the model training module; the data processing engine supports reading data from different data sources and performing mixing and preprocessing on data from different data sources.
[0031] The model training module is used to train the model based on the target slice data to obtain a model that meets the training termination conditions.
[0032] Thirdly, this disclosure provides an electronic device, including: a memory and a processor, the memory being configured to store computer program instructions; the processor being configured to execute the computer program instructions, causing the electronic device to implement the data processing method as described in the first aspect and any one of the first aspects.
[0033] Fourthly, this disclosure provides a readable storage medium, comprising: computer program instructions; at least one processor of an electronic device executes the computer program instructions, causing the electronic device to implement the data processing method as described in the first aspect and any one of the first aspects.
[0034] Fifthly, this disclosure provides a computer program product, which an electronic device runs to enable the electronic device to implement the data processing method as described in the first aspect and any one of the first aspects.
[0035] This disclosure provides a data processing method and apparatus. The method includes: calling a scheduling management module in a data processing system to slice the sample dataset corresponding to a model training task to obtain initial slice data; wherein the initial slice data includes sample data from one or more data sources; calling a data processing engine in the data processing system to perform mixing and / or preprocessing on the initial slice data to obtain target slice data that meets the data requirements of the model training module; and inputting the target slice data into the model training module for model training to obtain a model that meets the training termination conditions. By introducing a feature-rich data processing engine into the data processing system (model training framework) to perform rich-type sample data reading, mixing, and preprocessing, refined sample shuffling, mixing, and preprocessing functions can be achieved, enabling the model training framework to support training on sample data from various types of data sources and improving model training performance. Furthermore, in this disclosure, data preprocessing and model training can be processed in parallel, improving model training efficiency. Attached Figure Description
[0036] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.
[0037] To more clearly illustrate the technical solutions in the embodiments of this disclosure or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, those skilled in the art can obtain other drawings based on these drawings without creative effort.
[0038] Figure 1 This is a schematic diagram of the overall framework of a data processing system provided in an embodiment of the present disclosure;
[0039] Figure 2 A schematic flowchart illustrating a data processing method provided in an embodiment of this disclosure;
[0040] Figure 3 A schematic flowchart illustrating a data processing method provided in another embodiment of this disclosure;
[0041] Figure 4 A schematic flowchart illustrating a data processing method provided in another embodiment of this disclosure;
[0042] Figure 5 A schematic flowchart illustrating a data processing method provided in an embodiment of this disclosure;
[0043] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present disclosure. Detailed Implementation
[0044] To better understand the above-mentioned objectives, features, and advantages of this disclosure, the solutions disclosed herein will be further described below. It should be noted that, unless otherwise specified, the embodiments and features described herein can be combined with each other.
[0045] Numerous specific details are set forth in the following description in order to provide a full understanding of this disclosure, but this disclosure may also be implemented in other ways different from those described herein; obviously, the embodiments in the specification are only some, and not all, of the embodiments of this disclosure.
[0046] In offline model training scenarios, model training frameworks typically support sequential reading of files (sample data), and they have limitations on the data format of the sample data. For example, some model training frameworks can only read sample data in TFrecord format and cannot read other formats. TFrecord is a binary data format used by TensorFlow. Furthermore, some complex data preprocessing needs to be completed by a data preprocessing framework in the pre-stage. The data preprocessing framework triggers multiple preprocessing sub-processes, and the model training framework can only start after preprocessing is complete. It should be understood that traditional model training frameworks do not have data preprocessing capabilities.
[0047] The above methods have at least the following pain points:
[0048] 1. Current model training frameworks lack data mixing capabilities, such as row-level data mixing. Traditional methods read data sequentially from files, which cannot perform fine-grained sample shuffling or interactive logic, making it difficult to meet complex model training needs. For example, they cannot support fine-grained sample preprocessing and data mixing such as shuffle, groupBy, and repartition, thus limiting the diversity of sample data.
[0049] 2. It cannot support flexible and diverse data sources, especially when new data sources appear. Directly supporting multiple data source readings in each model training framework would lead to extremely high development costs.
[0050] 3. There are delays in data processing and training. Traditional methods require data preprocessing to be completed by the data preprocessing framework before training can begin. If there is a large amount of sample data and the preprocessing load is heavy, the waiting time to start training will be too long.
[0051] 4. Waste of computational resources. When training does not meet expectations, since the preprocessed sample data has already been generated offline, when the offline metrics of the model are abnormal and data needs to be regenerated, the data preprocessing framework needs to be called to perform data preprocessing again, and the training resources need to wait, which leads to a large waste of computational resources.
[0052] 5. Waste of storage resources. During offline training, it is usually necessary to persist a large amount of sample data, such as one or two years' worth of data. This data consumes a lot of storage resources, resulting in serious waste of storage resources.
[0053] To address the aforementioned issues, this disclosure provides a data processing method and apparatus. By introducing a feature-rich data processing engine into the data processing system to perform diverse sample data reading, data mixing, and preprocessing, it enables refined sample shuffling, mixing, and preprocessing functions. This allows the model training framework to support training on sample data from various data sources, improving model training performance. Furthermore, data mixing and preprocessing can be performed in parallel with model training, breaking the strict sequential dependency and improving model training efficiency.
[0054] For example, the data processing method provided in this disclosure can be implemented by a data processing device, which can be implemented in any software and / or hardware manner. For example, the data processing device can be, but is not limited to: tablet computers, mobile phones (such as foldable phones, large-screen phones, etc.), wearable devices, in-vehicle devices, laptops, ultra-mobile personal computers (UMPCs), netbooks, personal digital assistants (PDAs), servers, server clusters, cloud servers, etc. This disclosure does not impose any restrictions on the specific type of electronic device. The data processing device can also be a data processing system implemented in software, also known as a model training system.
[0055] The data processing method and data processing system provided in this disclosure will be described in detail below through several embodiments.
[0056] Figure 1 This is a schematic diagram of the overall framework of a data processing system provided according to an embodiment of this disclosure. Please refer to... Figure 1As shown, the data processing system 100 provided in this embodiment includes: a scheduling management module 101, one or more data processing engines 102, and a model training module 103. The one or more data processing engines 102 can be understood as part of the data preprocessing module.
[0057] The scheduling management module 101 is used to control the initiation of the model training task based on the received model training request, read the sample dataset for slicing to obtain initial slice data, and schedule the data processing engine 102 to perform mixed processing and preprocessing of sample data from different data sources in the initial slice data. In this disclosure, the model training task can be an offline model training task; therefore, the sample dataset can be offline sample data collected within a preset time period. The offline sample data can originate from multiple data sources, including, but not limited to, one or more of the following: a distributed file storage system (HDFS), S3 (Simple Storage Service), Kafka (a distributed publish-subscribe messaging system), and a feature platform (FeatureStore). The sample data contains several files, each containing several rows of samples; each row of samples contains multiple columns of feature information and labels. The sample data can also include some external extended data, such as data that cannot be directly stored in the data processing system, the latest feature information updated in real time, etc., which can be dynamically read in when model training is started.
[0058] The data processing engine 102 is primarily responsible for the mixed processing and / or preprocessing of sample data from different data sources in the initial input slice data, controlled by the scheduling management module 101. Specifically, the data processing engine 102 supports reading sample data in different formats from various data sources, thereby enriching the data sources for model training. Furthermore, the data processing engine 102 can break the limitation of reading sample data from data sources in file order, enabling the global scattering of samples from multiple model training modules 103.
[0059] Methods for mixing sample data include, but are not limited to, row-level data mixing and file mixing. Mixing primarily involves two operations: data aggregation and shuffling the data order of the sample data. Data aggregation refers to sorting sample data from multiple data sources by time and then aggregating the data based on the sorting results and predefined rules, such as aggregating sample data from different data sources by day or hour. Shuffling the data order refers to disrupting the order of the sample data within the initial slice. This prevents the data from remaining in a fixed order after entering the model training module, which could lead to the training process being easily affected by the physical storage order of the data. Shuffling can be done row-wise or file-wise. Random shuffling can also be used. The shuffling of the sample data order can be achieved by calling a predefined function.
[0060] Preprocessing can include, but is not limited to, information transformation, sample filtering, etc. Information transformation is used to transform the information in the sample data to extract the required features or to enhance the features. For example, it can calculate positive and negative examples based on the sample data, or generate new features by merging the features of the sample data in multiple dimensions. Sample filtering refers to filtering the samples. For example, dynamic negative example filtering can be used to discard some negative examples (70% of the negative examples) and allow some negative examples (30% of the negative examples) to enter the model training module based on information from multiple data sources and the negative example filtering ratio (e.g., a filtering ratio of 0.3), thereby improving the training speed.
[0061] It should be noted that the "data preprocessing" mentioned in this disclosure can be a hybrid process, preprocessing, or a combination of both. The specific operations included in data preprocessing may vary depending on the model training task.
[0062] The model training module 103 is mainly responsible for training the model based on the target slice data output by the data processing engine to obtain a model that meets the training conditions. The model training module 103 may include multiple trainers, which can train the same model to be trained from different dimensions, or they can train different models to be trained using the same sample dataset. This disclosure does not limit this.
[0063] The data processing engine 102 can add the processed target slice data to the data queue, and the model training module 103 reads the target slice data from the data queue and inputs it into the corresponding trainer for model training. In some embodiments, when the data processing engine 102 adds target slice data to the data queue, it can add it according to the order of the initial slice data obtained from the slice processing, ensuring that the model training module 103 can obtain the training data in the correct order.
[0064] Reference Figure 1 As shown in the architecture, the data processing engine 102 and the model training module 103 can be executed in parallel, which can break the strict sequential dependency between data preprocessing and model training in the traditional offline model training mode and improve the efficiency of model training.
[0065] exist Figure 1 Based on the illustrated embodiment, the initial slice data input to the data processing engine 102 can include two types of data: one type requires preprocessing, and the other does not require preprocessing and can be directly used by the model training module 103. These two types of data require different processing methods. Therefore, the data processing engine 102 can include two different data preprocessing modules: a first data processing module and a second data processing module. The first data processing module initiates a data preprocessing task and performs corresponding calculations on the sample data requiring preprocessing. The second data processing module performs copying, checking, and integrity verification on the sample data that does not require preprocessing. Then, the merging control module included in the data processing engine 103 packages the data output from the first and second data processing modules respectively, achieving a mixed alignment of the preprocessed stream and the direct stream, resulting in target slice data that can be delivered to the model training module 103.
[0066] In this disclosure, the data processing system 100 can provide multiple data processing engines 102 to concurrently perform preprocessing on multiple initial slice data. This disclosure does not limit the number of concurrent operations; it can be set according to the hardware environment in which the data processing system 100 is deployed and the offline model training requirements. Figure 1 The number of data processing engines in the illustrated embodiment can be set according to various requirements such as data preprocessing speed and sample dataset size.
[0067] exist Figure 1 Based on the illustrated embodiment, the data processing engine 102 performs preprocessing of the scheduling execution data based on the scheduling management module 101. Wherein, Figure 2 This illustration shows a scenario diagram illustrating how the scheduling management module 101 schedules the data processing engine 102. Please refer to... Figure 2 As shown, the scheduling management module 101 can construct a scheduling logic graph based on the model training task and multiple initial slice data obtained from data slicing, such as... Figure 2 As shown in the dashed box, the scheduling logic diagram includes multiple data preprocessing tasks that correspond one-to-one with multiple initial slice data, and the arrows indicate the hierarchical relationship between the multiple data preprocessing tasks. Then, the scheduling management module 101 schedules one or more data processing engines 102 to execute the data preprocessing tasks corresponding to the initial slice data based on the scheduling logic diagram.
[0068] During the execution of the data preprocessing task, the scheduling management module 101 will cyclically check the execution status of the first data processing module, the second data processing module, and the merging control module in the data processing engine 102, and determine whether the data preprocessing task is successful or unsuccessful based on the execution status of each module. If it fails, it needs to be handled according to the preset restart strategy.
[0069] The data preprocessing task is considered successful when all modules executing a data preprocessing task succeed; the entire data preprocessing task in the scheduling logic diagram is considered successful when all data preprocessing tasks in the scheduling logic diagram are successfully completed.
[0070] Figure 3 This is a schematic flowchart illustrating a data processing method provided in one embodiment of this disclosure. Please refer to [link / reference]. Figure 3 As shown, the method in this embodiment includes:
[0071] S101. The scheduling and management module in the data processing system is invoked to slice the sample dataset corresponding to the model training task to obtain initial slice data; the initial slice data includes sample data from one or more data sources.
[0072] The model training task indicates a task to train the model using a sample dataset. This task can be generated by the data processing system based on user-triggered operations. For example, the data processing system can display a visual interactive interface, allowing the user to input commands to instruct the system to perform model training. Upon receiving the user's input, the data processing system initiates the model training task. The user's input can include information about the model to be trained, the sample dataset, data granularity (such as the size of the time window, the number of initial slices, the size of the initial slices, etc., indicating the slice processing method), concurrency, and data preprocessing methods, among other things. Information not specified by the user can use the data processing system's default configuration.
[0073] The data processing system responds to user input to initiate model training tasks, reads the sample dataset, and performs slicing according to the data granularity specified by the user or using the system's default configuration. For example, if model training requires sample data collected over the past year, the year's sample data is sliced according to a preset time window (month) to obtain 12 initial slices.
[0074] Since offline tasks typically require collecting offline data over a relatively long period as training samples, after data is generated in the data source, it can be copied from the data source to the data storage module of the data processing system at preset intervals. Since different data sources may use different data formats, the data storage module of the data processing system supports data in various formats. When a model training task is received, sample data from each data source can be obtained from the data storage module of the data processing system. The data storage module of the data processing system can be a data storage system, such as HDFS.
[0075] S102. Call the data processing engine in the data processing system to perform mixed processing and / or preprocessing on the initial slice data to obtain target slice data that meets the data requirements of the model training module; the data processing engine supports reading data from different data sources and performing mixed processing and preprocessing on data from different data sources.
[0076] Combination Figure 1 and Figure 2 As shown, the scheduling management module generates a logical scheduling graph based on the initial slice data. Based on the logical scheduling graph and the number of concurrent requests, the corresponding number of data processing engines are called to perform data preprocessing on the initial slice data, and the resulting target slice data is added to the data queue.
[0077] If the model training has special requirements for the order of training data, a scheduling logic diagram can be constructed according to the specified order to control the order of data preprocessing and the order of target slice data in the data queue.
[0078] For example, if a year's sample data is sliced according to a preset time window (month), resulting in 12 initial slices, with a concurrency of 3, then 3 data processing engines are invoked to perform preprocessing sequentially according to the order of initial slices 1 to 12. Each of the 3 engines then processes initial slices 1 to 3. After a data processing engine finishes its task, it can add the resulting target slice data to the data queue and start preprocessing for initial slice 4. It's important to note that the order of the target slice data in the data queue must match the order of the initial slice data. If data processing engine 3 finishes preprocessing for initial slice 3 before data processing engines 1 and 2, then target slice 3 can only be added to the data queue after data processing engines 1 and 2 have completed their preprocessing and added target slices 1 and 2 to the data queue, ensuring the correct order of the target slice data in the queue.
[0079] If the model training does not require the order of the training data, the data processing engine can be scheduled to perform data preprocessing tasks based on the scheduling logic graph, and the corresponding target slice data can be added to the data queue according to the order in which the data processing engine completes the preprocessing.
[0080] Combination Figure 1 As shown, each data processing engine can process data in the following ways:
[0081] The data processing engine includes a first data processing module to preprocess the first data in the initial slice data to obtain third data that meets the data requirements of the model training module; the data processing engine includes a second data processing module to check whether the initial slice data includes second data that does not require preprocessing, and outputs the checked second data to the merging control module; the merging control module is then called to merge and package the third data output by the first data processing module and the second data output by the second data processing module and add them to the data queue.
[0082] S103. The target slice data is input into the model training module to train the model, so as to obtain a model that meets the training termination condition.
[0083] Each trainer in the model training module can load the model to be trained and input one or more target slice data read from the data queue into the model to be trained for training. Through continuous iterative training, the trained model is obtained until the training termination condition is met.
[0084] The model to be trained can be any model, such as an image processing model, a speech processing model, a text processing model, a classification model, and so on.
[0085] Training termination conditions may include, but are not limited to: all initial slice data have been trained, the model has reached the preset number of iterations, the model's accuracy meets specified conditions, etc.
[0086] In this embodiment, by introducing a feature-rich data processing engine into the data processing system to perform rich types of sample data reading, data mixing, and preprocessing, it is possible to achieve refined sample shuffling, mixing, and preprocessing functions, enabling the model training framework to support training on sample data from various types of data sources and improve the model training effect. In addition, in this disclosure, data preprocessing and model training can be processed in parallel, improving model training efficiency.
[0087] Figure 4 This is a schematic flowchart illustrating a data processing method provided in another embodiment of this disclosure. Please refer to [link / reference]. Figure 4 As shown, the method in this embodiment is Figure 3 Based on the illustrated embodiment, it also includes:
[0088] S104. Call the scheduling management module to check the execution status of the data processing engine.
[0089] S105. When the execution status of the data processing engine is detected to be in a failed state, the data processing engine is restarted based on a preset restart strategy to re-process and / or preprocess the corresponding initial slice data, or the processing and / or preprocessing of the corresponding initial slice data is terminated.
[0090] As mentioned above, each module in the data processing engine can report its execution status to the scheduling management module, enabling the scheduling management module to determine whether the execution status of the data processing engine is a failure or a success based on the obtained status information. In order to reduce the impact of data preprocessing failures on model training, a restart strategy (i.e., a preset restart strategy) can be deployed in the data processing system to handle data preprocessing tasks that fail to execute.
[0091] The preset restart strategy can be, but is not limited to, any of the following strategies: restart when the data processing engine fails, or determine whether to restart based on the maximum number of failures acceptable to the data processing system when the data processing engine fails.
[0092] In the latter implementation, when a data processing engine failure is detected, it is determined whether the failure has reached the maximum number of failures indicated by the preset restart strategy. If the maximum number of failures has not been reached, the data processing engine is restarted to reprocess the corresponding initial slice data. If the maximum number of failures has been reached, the preprocessing of the corresponding slice data is terminated, and the failure result is transmitted to the model training module. When the sample data volume is large and the data granularity is small, some sample data preprocessing errors can be allowed. Since this part of the sample data accounts for a small proportion, it has little impact on the model training effect and can therefore be ignored. Thus, when a data preprocessing failure is detected, the data preprocessing task does not need to be restarted. In some embodiments, the maximum number of failures indicated by the preset restart strategy can be determined based on the number of initial slice data obtained after splitting the sample dataset and a preset ratio, or it can be specified by the user. This disclosure does not limit this.
[0093] Users can choose any of the aforementioned appropriate methods to control the restart of the data preprocessing task based on the size of the sample dataset, data granularity, etc.
[0094] In this embodiment, the scheduling management module continuously monitors the execution status of the data processing engine. Based on the execution status, it ensures that failed data preprocessing tasks can be restarted, thereby ensuring the richness of sample data and improving the model training effect.
[0095] Optionally, in Figure 3 to Figure 4Based on the illustrated embodiment, to reduce storage resource waste, previously trained target slice data can be dynamically deleted during the training process. Specifically, the earliest trained target slice data can be deleted first, while retaining a portion of target slice data trained more recently, thereby reducing the storage resource consumption of target slice data.
[0096] Please see Figure 5 As shown, in a specific embodiment, the data processing system responds to user input instructions, extracts sample datasets from the data storage module of the data processing system, such as the distributed file system (Hadoop Distributed File System, HDFS), and creates a data stream; it also creates a trainer and a scheduling management module; the scheduling management module listens for the creation of the data stream, obtains the sample dataset for slicing processing, and generates a Directed Acyclic Graph (DAG), where DAG is one implementation of the scheduling logic graph described above; next, the DAG is executed, and the runtime of the data processing engine (such as Spark) can read the DAG information from the APIServer component of the scheduling management module to obtain the corresponding initial slice data, execute the corresponding data preprocessing tasks to obtain the processed target slice data, and save it to HDFS.
[0097] Based on the scheduling of the initial slice data, the target slice data in the shared storage is loaded sequentially, a new data stream is constructed (equivalent to implementing a data queue), training tasks are generated, and training data is delivered to the trainer in the model training module.
[0098] In addition, the scheduling management module records the execution status information reported by the data processing engine and records it in the DAG. The scheduling management module checks the execution status of each data preprocessing task recorded in the DAG and restarts it according to the preset restart strategy.
[0099] By way of example, this disclosure also provides an electronic device.
[0100] Figure 6 This is a schematic diagram of the structure of an electronic device provided according to an embodiment of this disclosure. Please refer to [link / reference]. Figure 6 As shown, the electronic device 600 provided in this embodiment includes a memory 601 and a processor 602.
[0101] The memory 601 can be a separate physical unit, connected to the processor 602 via a bus 603. Alternatively, the memory 601 and processor 602 can be integrated together, implemented in hardware, etc.
[0102] The memory 601 is used to store program instructions, and the processor 602 calls the program instructions to execute the data processing method provided in any of the above method embodiments.
[0103] Optionally, when some or all of the methods in the above embodiments are implemented by software, the electronic device 600 may also include only the processor 602. The memory 601 for storing programs is located outside the electronic device 600, and the processor 602 is connected to the memory via circuits / wires for reading and executing the programs stored in the memory.
[0104] The processor 602 can be a central processing unit (CPU), a network processor (NP), or a combination of a CPU and an NP.
[0105] The processor 602 may further include a hardware chip. This hardware chip may be an application-specific integrated circuit (ASIC), a programmable logic device (PLD), or a combination thereof. The PLD may be a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof.
[0106] The memory 601 may include volatile memory, such as random-access memory (RAM); the memory may also include non-volatile memory, such as flash memory, hard disk drive (HDD), or solid-state drive (SSD); the memory may also include combinations of the above types of memory.
[0107] This disclosure also provides a readable storage medium, including: computer program instructions, which, when executed by at least one processor of an electronic device, cause the electronic device to implement the data processing method provided in any of the above method embodiments.
[0108] This disclosure also provides a computer program product that, when run on a computer, enables the computer to implement the data processing method provided in any of the above method embodiments.
[0109] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.
[0110] The above description is merely a specific embodiment of this disclosure, enabling those skilled in the art to understand or implement it. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this disclosure. Therefore, this disclosure is not to be limited to the embodiments described herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. A data processing method, characterized in that, include: The scheduling and management module of the data processing system is invoked to slice the sample dataset corresponding to the model training task to obtain the initial slice data; The initial slice data includes sample data from one or more data sources; The data processing engine in the data processing system is invoked to perform mixing and / or preprocessing on the initial slice data to obtain target slice data that meets the data requirements of the model training module. The data processing engine supports reading data from different data sources and performing mixing and preprocessing on data from different data sources. The mixing process includes data aggregation and shuffling the data order of sample data. The mixing process is used to instruct the shuffling of the data order of sample data contained in the initial slice data. The preprocessing process is used to instruct information transformation and / or sample filtering on the initial slice data. The information transformation is used to transform the information in the sample data to extract the required features or perform feature enhancement. The target slice data is input into the model training module for model training to obtain a model that meets the training termination conditions.
2. The method according to claim 1, characterized in that, The step of invoking the data processing engine in the data processing system to perform mixing and / or preprocessing on the initial slice data includes: The data processing engine is invoked to sort the data in the one or more data sources according to time, and the sample data contained in the initial slice data is aggregated based on the sorting results; For each of the initial slice data, the data processing engine is invoked to shuffle the data order of the sample data contained in the initial slice data by line / by file to obtain the mixed initial slice data.
3. The method according to claim 1, characterized in that, The step of calling the data processing engine in the data processing system to perform mixed processing and / or preprocessing on the initial slice data to obtain target slice data that meets the data requirements of the model training module includes: The data processing engine is invoked to preprocess the first data in the initial slice data to obtain the third data that meets the data requirements of the model training module; and the initial slice data is checked to see if it includes second data that does not require preprocessing. The data processing engine is invoked to merge the third data with the second data obtained from the inspection that does not require preprocessing to obtain the target slice data, wherein the second data meets the data requirements of the model training module.
4. The method according to claim 1, characterized in that, The step of using the scheduling management module to call the data processing engine to perform mixed processing and / or preprocessing on the initial slice data to obtain target slice data that meets the data requirements of the model training module includes: Based on a preset concurrency level, multiple data processing engines in the data processing system are invoked in parallel to perform mixed processing and / or preprocessing on multiple initial slice data to obtain target slice data output by each of the data processing engines.
5. The method according to any one of claims 1 to 4, characterized in that, Also includes: The scheduling management module is invoked to check the execution status of the data processing engine; When the execution status of the data processing engine is detected to be in a failed state, the data processing engine is restarted based on a preset restart strategy to re-process and / or preprocess the corresponding initial slice data, or the processing and / or preprocessing of the corresponding initial slice data is terminated.
6. The method according to claim 5, characterized in that, The step of determining to restart the data processing engine based on a preset restart strategy to reprocess the corresponding initial slice data or to end the mixing and / or preprocessing of the corresponding initial slice data includes: Determine whether this failure has reached the maximum number of failures indicated by the preset restart policy; If the maximum number of failures is reached, the data processing engine is restarted to reprocess and / or preprocess the corresponding initial slice data. If the maximum number of failures is not reached, the mixing and / or preprocessing of the corresponding initial slice data is terminated, and the failure results are transmitted to the model training module.
7. The method according to claim 4, characterized in that, The step of inputting the target slice data into the model training module for model training includes: The target slice data output by the multiple data processing engines are sequentially input into the data queue according to the order in which the initial slice data are obtained through slice processing; The model training module is invoked to read the target slice data from the data queue for model training.
8. The method according to claim 1, characterized in that, Also includes: During the training process, the target slice data trained by the model training module is deleted from the data processing system.
9. A data processing apparatus, characterized in that, include: The scheduling and management module is used to slice the sample dataset corresponding to the model training task to obtain the initial slice data. The initial slice data includes sample data from one or more data sources; A data processing engine is used to perform mixing and / or preprocessing on the initial slice data to obtain target slice data that meets the data requirements of the model training module. The data processing engine supports reading data from different data sources and performing mixing and preprocessing on data from different data sources. The mixing process includes data aggregation and shuffling the data order of sample data. The mixing process is used to instruct the shuffling of the data order of each sample data contained in the initial slice data. The preprocessing process is used to instruct information transformation and / or sample filtering on the initial slice data. The information transformation is used to transform the information in the sample data to extract the required features or perform feature enhancement. The model training module is used to train the model based on the target slice data to obtain a model that meets the training termination conditions.
10. An electronic device, characterized in that, include: Memory and processor; The memory is configured to store computer program instructions; The processor is configured to execute the computer program instructions, causing the electronic device to implement the data processing method as described in any one of claims 1 to 8.
11. A readable storage medium, characterized in that, include: Computer program instructions; The electronic device executes the computer program instructions, causing the electronic device to implement the data processing method as described in any one of claims 1 to 8.
12. A computer program product, characterized in that, The electronic device runs the computer program product, causing the electronic device to implement the data processing method as described in any one of claims 1 to 8.