Configurable machine learning dataset preprocessing method, apparatus, device, and medium

Abstract

The application discloses a configured machine learning data set preprocessing method and device, equipment and medium, relates to the technical field of machine learning, and comprises the following steps: verifying and analyzing a data set preprocessing configuration file, replacing variable parameters in the configuration file, and generating a dependency directed graph representing data set preprocessing steps; calling a target main function based on the configuration file to load a target plug-in function in a preset plug-in algorithm pool; reading a target machine learning data set corresponding to the configuration file from a preset hierarchical storage directory, and performing parallel preprocessing on the data set according to the dependency directed graph through a thread pool, and saving the preprocessing result to the directory; during the preprocessing process, the execution state of each preprocessing step is recorded to facilitate breakpoint continuation, and the processing progress of each file data is continuously tracked to facilitate the tracking result being displayed to the front end, and when an exception is detected, the preset exception handling strategy is used for processing.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of machine learning, in particular to a configurable machine learning dataset preprocessing method, device, equipment and medium. BACKGROUND

[0002] In actual machine learning projects, data preprocessing usually accounts for more than 60% of the total workload. The common preprocessing schemes in the industry are as follows: script type, data scientists handwrite Python scripts for each dataset, flexible but not reusable; pipeline type, task dependencies are described through DAG (Directed Acyclic Graph), but the task granularity is coarse, and operators still need to be developed for each type of data; built-in platform, the platform has several fixed templates, and users can only choose, with poor flexibility. Therefore, scripts need to be written separately for each type of dataset, with redundant code, difficult maintenance, and the need to modify the main process code when adding new algorithms, making extension difficult. In addition, the processing progress, exception information, etc. in the preprocessing process are scattered in log files, making it difficult to track in real time, and the preprocessing stability is poor.

[0003] To sum up, how to optimize the machine learning dataset preprocessing method to improve the efficiency, stability and flexibility of dataset preprocessing is a technical problem to be solved at present. SUMMARY

[0004] Therefore, the purpose of the present application is to provide a configurable machine learning dataset preprocessing method, device, equipment and medium, which can optimize the machine learning dataset preprocessing method to improve the efficiency, stability and flexibility of dataset preprocessing. The specific scheme is as follows:

[0005] In a first aspect, the present application provides a configurable machine learning dataset preprocessing method applied to a dataset preprocessing system, comprising:

[0006] verifying and analyzing the received dataset preprocessing configuration file, and replacing variable parameters in the dataset preprocessing configuration file with values in the runtime context to generate a dependency relationship directed graph representing dataset preprocessing steps;

[0007] calling a target main function based on the dataset preprocessing configuration file to load a target plug-in function in a preset plug-in algorithm pool through the target main function; the target main function is any one of a text processing main function, a table processing main function and an image processing main function; the target plug-in function is an algorithm function registered based on a preset decorator and added to the preset plug-in algorithm pool;

[0008] reading a target machine learning dataset corresponding to the dataset preprocessing configuration file from a preset hierarchical storage directory, and performing parallel preprocessing on the target machine learning dataset according to the dependency directed graph through a thread pool, and saving the preprocessing result to the preset hierarchical storage directory;

[0009] During the preprocessing, the execution state of each preprocessing step is recorded to enable breakpoint continuation based on the execution state, and the processing progress of each file data in the target machine learning dataset is continuously tracked to enable the tracking result to be displayed to the front end of the dataset preprocessing system, and when an exception is detected, the preset exception handling strategy is used for processing.

[0010] Optionally, before the received dataset preprocessing configuration file is verified and parsed, the method further includes:

[0011] After the dataset preprocessing system is started, the preset plug-in algorithm pool is scanned to obtain function metadata of a plurality of plug-in functions in the preset plug-in algorithm pool;

[0012] The function metadata is counted to generate a corresponding plug-in function information list, and a function configuration form of the plurality of plug-in functions is generated according to the plug-in function information list;

[0013] The function configuration form is displayed to the front end to enable the dataset preprocessing configuration file to be obtained through the front end; the dataset preprocessing configuration file is a JSON file configured based on the function configuration form and a preset configuration file rule, and related information for preprocessing of the target machine learning dataset.

[0014] Optionally, the dataset preprocessing configuration file is verified and parsed, and a value in a runtime context is used to replace a variable parameter in the dataset preprocessing configuration file to generate a dependency directed graph representing a dataset preprocessing step, including:

[0015] The received dataset preprocessing configuration file is verified based on a target file verification tool and the plug-in function information list, and after verification, a preprocessing step list in the dataset preprocessing configuration file is determined;

[0016] The parameters of each preprocessing step in the preprocessing step list are analyzed to determine a corresponding precondition branch, and it is determined whether there is an unconfigured parameter dependent step to obtain a corresponding determination result;

[0017] If the judgment result represents that there is an unconfigured parameter dependent step, the parameter dependent step is inserted into the preprocessing step list to obtain an updated preprocessing step list, and a variable parameter in the updated preprocessing step list is identified;

[0018] The variable parameter is replaced by a value in a runtime context to generate a dependency directed graph representing data set preprocessing steps based on the obtained replaced preprocessing step list and the precondition branch.

[0019] Optionally, the target main function is called based on the data set preprocessing configuration file to load a target plug-in function in a preset plug-in algorithm pool through the target main function, including:

[0020] The data type configured in the data set preprocessing configuration file and the algorithm function name corresponding to each preprocessing step are determined;

[0021] If the data type is a text data type, the text processing main function is called to load a target plug-in function with the algorithm function name in a preset plug-in algorithm pool through the text processing main function;

[0022] If the data type is a table data type, the table processing main function is called to load a target plug-in function with the algorithm function name in a preset plug-in algorithm pool through the table processing main function;

[0023] If the data type is an image data type, the image processing main function is called to load a target plug-in function with the algorithm function name in a preset plug-in algorithm pool through the image processing main function.

[0024] Optionally, the preset hierarchical storage directory is a hierarchical storage directory based on a target object storage service, including a first folder, a second folder, and a third folder; the first folder is used to store original machine learning data sets to be preprocessed; the second folder is used to store intermediate processing results generated in the preprocessing process of the original machine learning data sets; and the third folder is used to store data preprocessing results obtained after preprocessing the original machine learning data sets.

[0025] Correspondingly, the target machine learning data set corresponding to the data set preprocessing configuration file is read from the preset hierarchical storage directory, and the target machine learning data set is parallelly preprocessed according to the dependency directed graph through a thread pool, and the preprocessing result is saved to the preset hierarchical storage directory, including:

[0026] Determine the parallelism configured in the dataset preprocessing configuration file, and read the target machine learning dataset corresponding to the dataset preprocessing configuration file from the first folder of the preset hierarchical storage directory;

[0027] Obtain the parameter verification model of the target plug-in function by using the parameter information acquisition function, and verify the variable parameters based on the parameter verification model to obtain a corresponding verification result;

[0028] If the verification result indicates that the variable parameters pass the verification, then according to the processing task type of the dependency relationship directed graph and the parallelism, the target machine learning dataset is parallel preprocessed by a thread pool to obtain a corresponding preprocessing result.

[0029] The preprocessing result is saved to the third folder, and the generated intermediate processing result is saved to the second folder during preprocessing.

[0030] Correspondingly, the parallel preprocessing of each file data in the target machine learning dataset includes:

[0031] If the data type of the target machine learning dataset is a table data type, the data in the target machine learning dataset is block read, and the target machine learning dataset is parallel preprocessed in a streaming processing manner.

[0032] If the data type of the target machine learning dataset is an image data type, a target caching algorithm is used to cache image decoding results, and the thumbnails of image data in the target machine learning dataset are detected for edges to determine the target key area of the image data, and the target key area is processed at full resolution.

[0033] Optionally, the processing according to the preset exception handling strategy when the exception is monitored includes:

[0034] If the variable parameters are monitored to be abnormal, the variable parameters are intercepted, and a corresponding target response status code is returned.

[0035] If it is monitored that the current data file to be processed is damaged, the data file to be processed is recorded, and the other unprocessed data files in the target machine learning dataset except the data file to be processed are parallel preprocessed.

[0036] If it is monitored that the currently executed plug-in algorithm crashes, the abnormal plug-in algorithm is retried, the corresponding retry number is recorded, and if the retry number reaches a preset retry number threshold, the current abnormal event is fed back.

[0037] If it is monitored that a software library or module relied on by the currently executed plug-in algorithm is missing, the missing software library or module is installed.

[0038] Optionally, the configured machine learning dataset preprocessing method further comprises:

[0039] A parameter verification model of the new algorithm function is defined using the target software library, and function meta information of the new algorithm function is determined;

[0040] The preset decorator is added in the code of the new algorithm function to obtain an updated new algorithm function;

[0041] The updated new algorithm function is added to the preset plug-in algorithm pool based on the function meta information and the parameter verification model, so that the new algorithm function is registered based on the preset decorator when the dataset preprocessing system is started.

[0042] In a second aspect, the present application provides a configured machine learning dataset preprocessing device applied to a dataset preprocessing system, comprising:

[0043] A parameter replacement module is configured to check and analyze the received dataset preprocessing configuration file, and replace variable parameters in the dataset preprocessing configuration file with values in a runtime context to generate a dependency directed graph representing dataset preprocessing steps;

[0044] A function loading module is configured to call a target main function based on the dataset preprocessing configuration file to load a target plug-in function in a preset plug-in algorithm pool through the target main function; the target main function is any one of a text processing main function, a table processing main function, and an image processing main function; the target plug-in function is an algorithm function registered based on a preset decorator and added to the preset plug-in algorithm pool;

[0045] A dataset preprocessing module is configured to read a target machine learning dataset corresponding to the dataset preprocessing configuration file from a preset hierarchical storage directory, and perform parallel preprocessing on the target machine learning dataset according to the dependency directed graph through a thread pool, and save the preprocessing result to the preset hierarchical storage directory.

[0046] A state recording module is configured to record an execution state of each preprocessing step in a preprocessing process, to resume the preprocessing process based on the execution state, to continuously track a processing progress of each file data in the target machine learning dataset, to display the tracking result to a front end of the dataset preprocessing system, and to process an exception according to a preset exception processing strategy when the exception is detected.

[0047] In a third aspect, the present application provides an electronic device, comprising:

[0048] A memory configured to store a computer program;

[0049] A processor configured to execute the computer program to implement the preceding configurable machine learning dataset preprocessing method.

[0050] In a fourth aspect, the present application provides a computer readable storage medium configured to store a computer program; wherein the computer program is executed by a processor to implement the preceding configurable machine learning dataset preprocessing method.

[0051] In the present application, the received dataset preprocessing configuration file is checked and parsed, and the variable parameters in the dataset preprocessing configuration file are replaced with the values in the runtime context to generate a dependency directed graph representing the dataset preprocessing steps; the target main function is called based on the dataset preprocessing configuration file to load the target plug-in function in the preset plug-in algorithm pool through the target main function; the target main function is any one of a text processing main function, a table processing main function, and an image processing main function; the target plug-in function is an algorithm function registered based on a preset decorator and added to the preset plug-in algorithm pool; the target machine learning dataset corresponding to the dataset preprocessing configuration file is read from a preset hierarchical storage directory, and the target machine learning dataset is parallelly preprocessed according to the dependency directed graph through a thread pool, and the preprocessing result is saved to the preset hierarchical storage directory; during preprocessing, the execution state of each preprocessing step is recorded to support breakpoint continuation based on the execution state, and the processing progress of each file data in the target machine learning dataset is continuously tracked to display the tracking result to the front end of the dataset preprocessing system, and when an exception is detected, the preset exception handling strategy is used for processing. As can be seen from the above, the received dataset preprocessing configuration file is checked and parsed, and the variable parameters in the file are replaced with the values in the runtime context to generate a dependency directed graph representing the preprocessing step dependencies, then the corresponding target main function is called according to the configuration file, the target plug-in function registered by the preset decorator is loaded in the preset plug-in algorithm pool through the target main function, then the target machine learning dataset corresponding to the configuration file is read from the preset hierarchical storage directory, the target machine learning dataset is parallelly preprocessed according to the dependency directed graph through the thread pool, and the preprocessing result is saved to the preset hierarchical storage directory. At the same time, the execution state of each step is recorded during preprocessing to support breakpoint continuation, the processing progress of each file data is continuously tracked and displayed to the front end of the system, and when an exception is detected, the preset strategy is used for processing. In this way, through the above process of the present application, the checking and parsing of the configuration file and the replacement of the variables ensure the accuracy and adaptability of the preprocessing rules, the dependency directed graph is generated, the logic between the steps is sorted out, and a reliable basis is provided for parallel processing; the calling of the corresponding main function and plug-in function enhances the flexibility and expandability of preprocessing, which can adapt to the processing needs of different types of data; the thread pool parallel processing combined with the read-write management of the hierarchical storage directory improves the preprocessing efficiency and avoids data storage chaos; the execution state recording, progress tracking, and exception handling mechanism not only guarantee the stability of the processing process, but also make it easy for users to continue processing after interruption, and enable users to keep abreast of the progress and respond to problems in a timely manner, which can optimize the machine learning dataset preprocessing method to improve the efficiency, stability, and flexibility of dataset preprocessing. BRIEF DESCRIPTION OF DRAWINGS

[0052] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or prior art description will be briefly introduced. Obviously, the drawings in the following description only represent some embodiments of the present application, and other drawings can be obtained by those skilled in the art without any creative effort on the basis of the provided drawings.

[0053] Figure 1 A configuration machine learning data set preprocessing method flow chart disclosed by the present application;

[0054] Figure 2 A configuration machine learning data set preprocessing method flow chart disclosed by the present application;

[0055] Figure 3 A configuration machine learning data set preprocessing method flow chart disclosed by the present application;

[0056] Figure 4 A configuration machine learning data set preprocessing method flow chart disclosed by the present application; DETAILED DESCRIPTION

[0057] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments only represent some embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without any creative effort belong to the scope of protection of the present application.

[0058] The current common preprocessing scheme is as follows: script type, data scientists handwrite Python scripts for each data set, flexible but not reusable; pipeline type, task dependency is described through DAG, but the task granularity is coarse, and operators still need to be developed for each type of data; built-in platform, the platform has several fixed templates, and users can only choose, which has poor flexibility. Therefore, scripts need to be written separately for each type of data set, which has code redundancy and maintenance difficulties. At the same time, new algorithms need to modify the main process code, which has extension difficulties, and parameters are hard-coded into the code, which requires re-deployment for changes. In addition, the processing progress, abnormal information and the like in the preprocessing process are scattered in the log file, which is difficult to realize real-time tracking, and the preprocessing stability is poor.

[0059] In order to overcome the above technical problems, the present application provides a configuration machine learning data set preprocessing method, which can optimize the machine learning data set preprocessing method to improve the efficiency, stability and flexibility of data set preprocessing.

[0060] Reference Figure 1As shown, the embodiment of the application discloses a configurable machine learning data set preprocessing method, applied to a data set preprocessing system, comprising:

[0061] In step S11, the received data set preprocessing configuration file is checked and parsed, and the variable parameters in the data set preprocessing configuration file are replaced with the values in the runtime context to generate a dependency directed graph representing the data set preprocessing steps.

[0062] In this embodiment, the data set preprocessing system first receives a predefined data set preprocessing configuration file, checks and parses the data set preprocessing configuration file to ensure the legality, integrity and readability of the configuration file, and then replaces the variable parameters contained in the configuration file with the actual values in the runtime context to realize parameter binding, and finally generates a dependency directed graph representing the dependency relationship between the data set preprocessing steps. For example Figure 2 As shown is a configurable machine learning data set preprocessing method flowchart provided by the present application.

[0063] It needs to be explained that before receiving the dataset preprocessing configuration file, system initialization needs to be carried out, and the processing flow is as follows: after the start of the dataset preprocessing system, the preset plug-in algorithm pool is scanned to obtain the function meta information of a plurality of plug-in functions in the preset plug-in algorithm pool; the function meta information is counted to generate a corresponding plug-in function information list, and a function configuration form of the plurality of plug-in functions is generated according to the plug-in function information list; the function configuration form is displayed to the front end, so as to obtain the dataset preprocessing configuration file through the front end; the dataset preprocessing configuration file is a JSON file configured for preprocessing related information of the target machine learning dataset based on the function configuration form and a preset configuration file rule. Among them, the plug-in function information list includes the function signature (parameter name, type, default value) of the plurality of plug-in functions, the supported dataType (data type) list, and the performance index (average time consumption of a single image); the preset configuration file rule is a predefined unified JSON configuration file specification. That is, after the start of the dataset preprocessing system, the preset plug-in algorithm pool, that is, the plugins / directory, is scanned first, the plugins / directory is used to save the plurality of plug-in functions and the corresponding meta information thereof, the function meta information of the plurality of plug-in functions is obtained, then the function meta information is counted and arranged to generate a corresponding plug-in function information list plugin_manifest.json, and the function configuration form of the plurality of plug-in functions is generated according to the list, then the function configuration form is displayed on the front end interface of the dataset preprocessing system, so that the user configures the preprocessing related information of the target machine learning dataset based on the function configuration form and the preset configuration file rule through the front end, finally generates a dataset preprocessing configuration file stored in JSON format, so that the dataset preprocessing system obtains the dataset preprocessing configuration file through the front end. Table 1 below is a preset configuration file rule schematic table of the dataset preprocessing configuration file.

[0064] Table 1 Preset configuration file rule schematic table

[0065]

[0066] The following is a configuration example of the processing steps:

[0067] {

[0068] "method": "canny",

[0069] "threshold1": 50,

[0070] "threshold2": 150,

[0071] "colorSpace": "GRAY",

[0072] "precondition": "width>256" / / support precondition expression

[0073] };

[0074] Wherein, the method represents the name of the specified pre-processing or feature engineering algorithm, corresponding to a plug-in function; threshold1 and threshold2 are parameters required by the algorithm; colorSpace indicates the color space of image processing; precondition represents the precondition for executing the step.

[0075] Specifically, the received dataset preprocessing configuration file is checked based on a target file checking tool and the plugin function information list, and after the checking passes, a preprocessing step list in the dataset preprocessing configuration file is determined; parameters of each preprocessing step in the preprocessing step list are analyzed to determine corresponding precondition branches, and it is judged whether there is an unconfigured parameter dependent step to obtain a corresponding judgment result; if the judgment result represents that there is an unconfigured parameter dependent step, the parameter dependent step is inserted into the preprocessing step list to obtain an updated preprocessing step list, and variable parameters in the updated preprocessing step list are identified; values in a runtime context are used to replace the variable parameters to generate a dependency relationship directed graph representing dataset preprocessing steps based on the obtained replacement preprocessing step list and the precondition branches. That is, first, the dataset preprocessing configuration file is checked by means of the target file checking tool and the plugin function information list, for example, the JSON legality is verified using the jsonschema library (a software library for verifying the structure and consistency of JOSN data), and after the checking passes, the preprocessing step list in the configuration file is determined, then the parameters of each preprocessing step in the list are analyzed, the precondition field is parsed, the precondition branches corresponding to each step are determined, and it is judged whether there is an unconfigured parameter dependent step, if there is, the parameter dependent step is inserted into the preprocessing step list to form an updated preprocessing step list, for example, canny (an edge detection algorithm) depends on the grayscale color space, that is, colorSpace=GRAY, so the convert (RGB2GRAY) (a function for converting data types) step is automatically inserted, then the variable parameters in the updated list are identified, finally the variable parameters are replaced by actual values in the runtime context, such as ${dataset_id} is dynamically replaced by the actual value, and based on the replacement preprocessing step list and the precondition branches, a dependency relationship directed graph capable of representing the correlation between each step of dataset preprocessing is generated.In this way, the verification process of the configuration file in the embodiment can check the file configuration problem in advance, avoiding the error of the subsequent preprocessing process caused by the defects in the file itself; the parsing process converts the abstract rules in the configuration file into executable step logic, ensuring that the system accurately understands the preprocessing requirements; the variable parameters are replaced by the runtime context value, making the preprocessing steps in the configuration file adapt to the actual running environment, avoiding the deviation of the preprocessing result caused by undefined variables or unmatched values, and improving the accuracy of preprocessing; the generated dependency relationship directed graph can intuitively present the sequence and dependency relationship of each preprocessing step, facilitating the operator to master the overall logic of the preprocessing process and providing a visual basis for subsequent preprocessing operation; the system actively scans the plugin algorithm pool and obtains function meta information when starting, which can master the currently available plugin function resources in real time and ensure that the functions called in the subsequent preprocessing configuration are valid and up-to-date; the configuration file adopts the JSON format, ensuring the standardization and uniformity of the configuration file, and the JSON format has good compatibility and readability, improving the efficiency and reliability of the data set preprocessing configuration link; the detection and supplement of the unconfigured parameter dependent step can perfect the preprocessing process chain, prevent the failure of the subsequent step caused by the lack of key dependent steps, and ensure the integrity of the preprocessing process.

[0076] In step S12, a target main function is called based on the data set preprocessing configuration file to load a target plugin function in a preset plugin algorithm pool through the target main function; the target main function is any one of a text processing main function, a table processing main function and an image processing main function; and the target plugin function is an algorithm function registered based on a preset decorator and added to the preset plugin algorithm pool.

[0077] In the embodiment, the corresponding target main function is called based on the data set preprocessing configuration file, and the target plugin function in the preset plugin algorithm pool is loaded through the target main function. The target main function is any one of a text processing main function, a table processing main function and an image processing main function; and the target plugin function is an algorithm function added to the preset plugin algorithm pool after being registered through a preset decorator.

[0078] It should be noted that the target main function is called based on the data set preprocessing configuration file, and the processing flow of loading the target plug-in function in the preset plug-in algorithm pool through the target main function is as follows: determining the data type configured in the data set preprocessing configuration file and the algorithm function name corresponding to each preprocessing step; if the data type is a text data type, the text processing main function is called to load the target plug-in function with the algorithm function name in the preset plug-in algorithm pool through the text processing main function; if the data type is a table data type, the table processing main function is called to load the target plug-in function with the algorithm function name in the preset plug-in algorithm pool through the table processing main function; if the data type is an image data type, the image processing main function is called to load the target plug-in function with the algorithm function name in the preset plug-in algorithm pool through the image processing main function. That is, first, the data type configured in the data set preprocessing configuration file and the algorithm function name corresponding to each preprocessing step are determined, and then differentiated calling operations are performed according to the difference of the data type, and the routing table of the target main function calling is as follows:

[0079] textDataset → text_preprocess.py:text_main()

[0080] tableDataset → table_preprocess.py:table_main()

[0081] picDataset → image_preprocess.py:image_main();

[0082] Specifically, if the data type is a text data type, a text processing main function is called to load a target plug-in function in the preset plug-in algorithm pool with a name consistent with the algorithm function name; if the data type is a table data type, a table processing main function is called to load a target plug-in function in the plug-in algorithm pool with a corresponding name; and if the data type is an image data type, an image processing main function is called to load a target plug-in function in the plug-in algorithm pool with a matching name. It needs to be further pointed out that the embodiment can dynamically load modules on demand by importlib.import_module(), a function module for dynamically importing a module at runtime, to avoid full loading at startup. In this way, the embodiment can call a target main function of a specific type according to a configuration file, accurately match the function call with the specific needs of the data set preprocessing (such as processing text, table or image data), and avoid function mismatch; the target main function serves as a bridge for loading plug-in functions, simplifies the plug-in calling process, does not need to design calling logic for each plug-in function separately, improves the convenience and uniformity of function calling, and adopts a preset decorator to register the target plug-in function and include it in the plug-in algorithm pool, which not only reduces the technical threshold for plug-in integration, enables new algorithm functions to be quickly and normatively added to the system, enhances the extensibility and flexibility of the plug-in algorithm pool, but also ensures the format compliance of the plug-in functions through the unified management of the decorator, and guarantees the stability, adaptability and scalability of function calling in the data set preprocessing process.

[0083] In step S13, the target machine learning data set corresponding to the data set preprocessing configuration file is read from the preset hierarchical storage directory, and the target machine learning data set is parallelly preprocessed according to the dependency directed graph through a thread pool, and the preprocessing result is saved to the preset hierarchical storage directory.

[0084] In the embodiment, first, the target machine learning data set corresponding to the data set preprocessing configuration file is read from the preset hierarchical storage directory, then the target machine learning data set is parallelly preprocessed according to the dependency directed graph generated in advance by using a thread pool, and finally the completed preprocessing result is saved to the preset hierarchical storage directory.

[0085] It needs to be explained that the preset hierarchical storage directory is a hierarchical storage directory based on a target object storage service, including a first folder, a second folder and a third folder; the first folder is used to store original machine learning data sets to be preprocessed; the second folder is used to store intermediate processing results generated in the preprocessing process of the original machine learning data sets; and the third folder is used to store data preprocessing results obtained after preprocessing the original machine learning data sets. Wherein, the target object storage service can be MinIO (an object storage service, a data storage center). As follows is a directory structure example of the preset hierarchical storage directory:

[0086] / datasets / {dataset_id} / v{version_id} /

[0087] ├── raw / # original file

[0088] ├── processed / # intermediate result

[0089] └── featured / # final feature

[0090] Wherein, v is a version identifier; └── represents a sibling directory; and └── represents the last subdirectory of the current level. The raw is the first folder; the processed is the second folder; and the featured is the third folder.

[0091] Specifically, the parallelism configured in the dataset preprocessing configuration file is determined, and the target machine learning dataset corresponding to the dataset preprocessing configuration file is read from the first folder of the preset hierarchical storage directory; the parameter verification model of the target plug-in function is obtained by using a parameter information acquisition function, and the variable parameters are checked based on the parameter verification model to obtain a corresponding checking result; if the checking result represents that the variable parameters pass the check, then according to the processing task type of the dependency relationship directed graph and the parallelism, the thread pool is used to perform parallel preprocessing on each file data in the target machine learning dataset to obtain a corresponding preprocessing result; the preprocessing result is saved to the third folder, and in the preprocessing process, the generated intermediate processing result is saved to the second folder. That is, first, the parallelism configured in the dataset preprocessing configuration file is determined, and the target machine learning dataset corresponding to the configuration file is read from the first folder of the preset hierarchical storage directory, then the parameter verification model of the target plug-in function is obtained by using a parameter information acquisition function such as inspect.signature (a function for obtaining signature information of a callable object), the variable parameters are checked by using the model, if the check passes, then according to the processing task type of the dependency relationship directed graph and the parallelism, the thread pool is used to perform parallel preprocessing on each file data in the target machine learning dataset, for example, the CPU (Central Processing Unit) intensive (such as PCA (principal component analysis)) and IO (input / output) intensive (such as MinIO read / write) tasks are mixedly scheduled by using the thread pool class method concurrent.futures.ThreadPoolExecutor of the module, the CPU intensive task consumes a large amount of resources, the threads can be reasonably allocated according to the core number, the IO intensive task mostly waits for read / write operations, more threads can be allocated, when waiting for IO, other threads continue to process data, and finally the preprocessing result is obtained, the preprocessing result is saved to the third folder of the preset hierarchical storage directory, and in the preprocessing process, the generated intermediate processing result is saved to the second folder in real time.

[0092] It should be noted that in order to realize performance optimization, the embodiment can perform algorithm-level optimization in the process of parallel preprocessing of the target machine learning dataset, and the processing flow is as follows: if the data type of the target machine learning dataset is a table data type, the data in the target machine learning dataset is read in blocks, and the target machine learning dataset is parallelly preprocessed in a streaming processing manner; if the data type of the target machine learning dataset is an image data type, the image decoding result is cached by using a target caching algorithm, and the edge of the thumbnail of the image data in the target machine learning dataset is detected to determine the target key area of the image data, and the target key area is processed at full resolution. That is, if the data type of the target machine learning dataset is a table data type, the table files in the target machine learning dataset are read in blocks by using pandas.read_csv(chunksize=10000) to carry out parallel preprocessing in a streaming processing manner, so as to avoid OOM (out of memory); if the data type is an image data type, the image decoding result of the last 10 images is cached by using a target caching algorithm, such as LRUCache (a kind of caching algorithm), to reduce repeated IO, then the edge of the thumbnail (256x256) of the image data in the target machine learning dataset is detected to determine the target key area (ROI) of the image data, and finally the target key area is processed at full resolution. In addition, for the image data type, cv2.cudaGPU acceleration can also be used in the processing process to improve the data preprocessing efficiency.In this way, the preset hierarchical storage directory of the embodiment provides a structured storage carrier for reading and result saving of the data set, facilitates quick positioning and acquisition of the target data set, improves data search efficiency, and completely separates the original data, intermediate data and result data in storage, avoids data confusion, and facilitates subsequent individual management, tracing and reuse of data at different stages; the parallel preprocessing based on the dependency relationship directed graph is implemented by means of the thread pool, which can fully utilize thread resources, improve processing efficiency, and at the same time ensure that the processing tasks between threads will not be executed in disorder due to dependency conflicts, thereby guaranteeing the orderliness and accuracy of the preprocessing process; the variable parameters are checked by the parameter verification model, which can check parameter abnormal problems in advance and guarantee the stability of the preprocessing process; the parallel preprocessing mode combining block reading and stream processing is adopted for the table data type, which can effectively solve the problem of excessive memory occupation caused by one-time reading of large-scale table data, reduce the system memory load, and at the same time realize the synchronous advancement of data reading and preprocessing through stream processing, thereby further improving the efficiency of parallel preprocessing; for image data type, the target cache algorithm is used to cache image decoding results, which can avoid repeated decoding of the same image and reduce the waste of computing resources; through the multi-scale processing mode of first performing edge detection on the thumbnail to determine the target key area and then performing full-resolution processing on the key area, the processing efficiency can be improved while guaranteeing the quality of image data preprocessing.

[0093] In step S14, the execution state of each preprocessing step is recorded during the preprocessing process, so as to perform breakpoint continuation based on the execution state, and continuously track the processing progress of each file data in the target machine learning data set, so as to display the tracking result to the front end of the data set preprocessing system, and process according to the preset abnormal processing strategy when an abnormality is monitored.

[0094] In this embodiment, during the execution of preprocessing, on the one hand, the execution state of each preprocessing step is recorded to provide data support for the subsequent breakpoint resume function that may be needed; on the other hand, the processing progress of each file data in the target machine learning data set is continuously tracked. That is, a ProgressEvent (an event object containing detailed progress information) is generated for each file, including step_name (step name), file_index (file index), total_files (total number of files), error_msg (error information), and the tracked progress result is pushed to the front-end interface in real time through websocket (a full-duplex communication protocol) so that the user can view the processing progress in real time on the front-end interface; at the same time, if an exception occurs in the preprocessing process, it is handled in a timely manner according to the preset exception handling strategy. The recording operation of the execution state is specifically as follows: after completing each preprocessing step, the current state is written into a checkpoint.json file, and the following is an example of step state:

[0095] {

[0096] "current_step": 3,

[0097] "completed_files": ["file1.jpg", "file2.jpg"],

[0098] "failed_files": ["file3.jpg"]

[0099] };

[0100] Wherein, current_step represents the execution order of the step being recorded; completed_files represents the data files that have completed preprocessing; failed_files represents the data files that have not completed preprocessing.

[0101] It should be noted that the processing procedure of processing according to the preset abnormality processing strategy when the abnormality is monitored to occur is as follows: if the variable parameter abnormality is monitored, the variable parameter is intercepted, and the corresponding target response status code is returned; if the current to-be-processed data file is monitored to be damaged, the to-be-processed data file is recorded, and the other unprocessed data files in the target machine learning data set except the to-be-processed data file are preprocessed in parallel; if the currently executed plug-in algorithm is monitored to crash, the abnormal plug-in algorithm is retried to execute, and the corresponding retry number is recorded, if the retry number reaches a preset retry number threshold, the current abnormal event is fed back; if the software library or module relied on by the currently executed plug-in algorithm is monitored to be missing, the missing software library or module is installed. That is, if the variable parameter is monitored to be abnormal, the variable parameter is immediately intercepted, and the corresponding target response status code is returned, for example, 400 Bad Request; if the current to-be-processed data file is found to be damaged, the damaged to-be-processed data file is recorded first, and the other unprocessed data files in the target machine learning data set except the damaged file are continuously preprocessed in parallel; if the currently executed plug-in algorithm is monitored to crash, the abnormal plug-in algorithm is retried to execute, and the number of each retry is recorded, when the retry number reaches a preset retry number threshold, the current abnormal event is fed back for manual intervention; if the software library or module relied on by the currently executed plug-in algorithm is monitored to be missing, the missing software library or module is automatically installed. The processing strategy example table of the abnormal processing event is as follows:

[0102] Table 2 Processing strategy example table of abnormal processing event

[0103]

[0104] It should be further pointed out that the embodiment can adopt a zero-copy transmission mechanism in the data preprocessing process, specifically, shared_memory is used between MinIO and the Python process, that is, the same block of memory area is shared, to avoid serialization overhead. At the same time, 100 small files are combined into 1 tar (compressed) package for uploading, to reduce the number of HTTP (HyperText Transfer Protocol) requests.

[0105] It needs to be explained that the processing flow when algorithm expansion is needed is as follows: the parameter verification model of the newly added algorithm function is defined by using the target software library, and the function meta information of the newly added algorithm function is determined; the preset decorator is added in the code of the newly added algorithm function to obtain the updated newly added algorithm function; the updated newly added algorithm function is added to the preset plug-in algorithm pool based on the function meta information and the parameter verification model, so that when the data set preprocessing system starts, the newly added algorithm function is registered based on the preset decorator. That is, first, with the help of the target software library, such as pydantic.BaseModel (a software library for data verification and management), the parameter verification model corresponding to the newly added algorithm function is defined, and the following is an example of a parameter verification model:

[0106] class CannyParams(BaseModel):

[0107] threshold1: int = Field(ge=0, le=255)

[0108] threshold2: int = Field(ge=0, le=255);

[0109] At the same time, the function meta information of the newly added algorithm function is determined, and then the preset decorator, such as @register_plugin(domain, name), is added in the code of the newly added algorithm function, domain is the processing domain, and name is the algorithm name, to form the updated newly added algorithm function, for example:

[0110] @register_plugin("image", "canny")

[0111] def canny(image_list, threshold1, threshold2): ...;

[0113] Finally, according to the function meta information and the parameter verification model that have been determined, the updated newly added algorithm function is added to the preset plug-in algorithm pool, so that when the data set preprocessing system starts, the registration of the newly added algorithm function can be completed based on the preset decorator. The following is a benchmark test result table of a data set preprocessing method provided by the present application.

[0114] Table 3 Benchmark test result table of data set preprocessing method

[0115]

[0116] In this way, the embodiment records the state of each preprocessing step, can retain the specific position and state information when the process is interrupted, provides support for resuming from the breakpoint, greatly reduces the repeated workload, and improves the fault tolerance and efficiency of the process; continuously tracking the file data processing progress and displaying it on the front end, enabling users to keep abreast of the overall advancement of preprocessing in real time, and enhancing the transparency of the process; the timely application of the preset abnormality processing strategy can quickly respond when an abnormality occurs, ensuring the stability and data security of the preprocessing process; using the target software library definition parameter verification model can ensure that the parameter verification logic of the newly added algorithm function conforms to the unified standard of the system, ensuring the accuracy of function calling; adding a preset decorator in the code makes the newly added function conform to the unified specification of system plug-in registration, ensuring that the system can automatically identify and complete registration when starting, without additional development of registration logic, reducing the technical threshold of function integration, and further enhancing the practicality and flexibility of the system.

[0117] As can be seen from the above, the embodiment of the present application checks and analyzes the received data set preprocessing configuration file, and replaces the variable parameters in the file with the values in the runtime context to generate a directed graph representing the dependency relationship of the preprocessing steps. Then, the corresponding target main function is called according to the configuration file, the target plug-in function registered by the preset decorator in the preset plug-in algorithm pool is loaded through the target main function, and then the target machine learning data set corresponding to the configuration file is read from the preset hierarchical storage directory. The target machine learning data set is parallelly preprocessed by means of the thread pool and according to the dependency relationship directed graph, and the preprocessing result is saved to the preset hierarchical storage directory. At the same time, the execution state of each step is recorded during the preprocessing process to support breakpoint continuation, the processing progress of each file data is continuously tracked and displayed to the system front end, and the preset strategy is processed when an exception is monitored. In this way, through the above process of the embodiment of the present application, on the one hand, the checking and processing of the configuration file can check the file configuration problem in advance, avoiding errors in the subsequent preprocessing process due to defects in the file itself. On the one hand, the variable parameters are replaced with the runtime context values, so that the preprocessing steps in the configuration file are adapted to the actual running environment, avoiding deviation of the preprocessing result and improving the accuracy of the preprocessing. On the one hand, the dependency relationship directed graph is generated, which can intuitively present the sequence and dependency relationship of each preprocessing step, making it easy for the operator to master the overall logic of the preprocessing process and providing a visual basis for subsequent preprocessing operations. On the one hand, the system actively scans the plug-in algorithm pool and obtains the function meta information when starting, which can master the currently available plug-in function resources in real time, ensuring that the functions called by the subsequent preprocessing configuration are valid and up-to-date. On the one hand, the configuration file adopts JSON format, which ensures the standardization and uniformity of the configuration file. JSON format has good compatibility and readability, which improves the efficiency and reliability of the data set preprocessing configuration link. On the one hand, the detection and supplement of the unconfigured parameter dependent steps can perfect the preprocessing process chain, prevent the failure of the subsequent steps due to the lack of key dependent steps, and ensure the integrity of the preprocessing process. On the one hand, the target main function of a specific type is called according to the configuration file, which can accurately match the function call with the specific needs of the data set preprocessing (such as processing text, table or image data), avoiding function mismatch. On the one hand, the target main function serves as a bridge for loading plug-in functions, simplifying the plug-in calling process, eliminating the need to design calling logic for each plug-in function, and improving the convenience and uniformity of function calling. On the one hand, the target plug-in function is registered by the preset decorator and included in the plug-in algorithm pool, which not only reduces the technical threshold of plug-in integration, enables new algorithm functions to be quickly and normatively added to the system, enhances the extensibility and flexibility of the plug-in algorithm pool, but also ensures the format compliance of the plug-in function through the unified management of the decorator, ensuring the stability, adaptability and scalability of the function calling in the data set preprocessing process.On the one hand, the preset hierarchical storage directory provides a structured storage carrier for data set reading and result saving, facilitates quick positioning and acquisition of target data set, improves data search efficiency, and at the same time completely separates original data, intermediate data and result data in storage, avoids data confusion, and facilitates subsequent separate management, tracing and reuse of data at different stages; on the one hand, parallel preprocessing based on a directed graph of dependency relationships is realized with the help of a thread pool, which can fully utilize thread resources, improve processing efficiency, and at the same time ensure that processing tasks between threads will not be executed in disorder due to dependency conflicts, ensuring the orderliness and accuracy of the preprocessing process; on the one hand, variable parameters are checked by a parameter verification model, which can detect parameter abnormality problems in advance and ensure the stability of the preprocessing process; on the one hand, for table data types, a parallel preprocessing mode combining block reading and streaming processing is adopted, which can effectively solve the problem of excessive memory occupation caused by one-time reading of large-scale table data, reduce system memory load, and at the same time realize the synchronous advancement of data reading and preprocessing through streaming processing, further improving the efficiency of parallel preprocessing; on the one hand, for image data types, the target cache algorithm is used to cache image decoding results, which can avoid repeated decoding of the same image and reduce waste of computing resources; on the one hand, through edge detection of thumbnails to determine the target key area, and then performing full-resolution processing on the key area, a multi-scale processing mode can improve processing efficiency while ensuring the quality of image data preprocessing; on the one hand, the execution state of each preprocessing step is recorded, which can retain the specific position and state information when the process is interrupted, provide support for breakpoint continuation, greatly reduce the amount of repeated work, and improve the fault tolerance and efficiency of the process; on the one hand, the file data processing progress is continuously tracked and displayed on the front end, so that users can master the overall advancement of preprocessing in real time, enhancing the transparency of the process; on the one hand, the timely application of preset exception handling strategies can quickly respond when an exception occurs, ensuring the stability and data security of the preprocessing process; on the one hand, the target software library is used to define the parameter verification model, which can ensure that the parameter checking logic of the newly added algorithm function conforms to the unified standard of the system, ensuring the accuracy of function calling; on the one hand, a preset decorator is added to the code, so that the newly added function conforms to the unified specification of system plug-in registration, ensuring that the system can automatically identify and complete registration when starting, without additional development of registration logic, reducing the technical threshold of function integration, further enhancing the practicality and flexibility of the system, and thus optimizing the machine learning data set preprocessing method to improve the efficiency, stability and flexibility of data set preprocessing.

[0118] Correspondingly, referring to Figure 3 The embodiment of the application also provides a configured machine learning data set preprocessing device applied to a data set preprocessing system, which comprises:

[0119] The parameter replacement module 11 is configured to check and parse the received data set preprocessing configuration file, and replace variable parameters in the data set preprocessing configuration file with values in the runtime context, to generate a dependency directed graph representing data set preprocessing steps.

[0120] The function loading module 12 is configured to call a target main function based on the data set preprocessing configuration file, to load a target plug-in function in a preset plug-in algorithm pool through the target main function; the target main function is any one of a text processing main function, a table processing main function, and an image processing main function; and the target plug-in function is an algorithm function registered based on a preset decorator and added to the preset plug-in algorithm pool.

[0121] The data set preprocessing module 13 is configured to read a target machine learning data set corresponding to the data set preprocessing configuration file from a preset hierarchical storage directory, and perform parallel preprocessing on the target machine learning data set according to the dependency directed graph through a thread pool, and save a preprocessing result to the preset hierarchical storage directory.

[0122] The state recording module 14 is configured to record an execution state of each preprocessing step in a preprocessing process, to perform breakpoint continuation based on the execution state, and continuously track a processing progress of each file data in the target machine learning data set, to display a tracking result to a front end of the data set preprocessing system, and to perform processing according to a preset exception handling strategy when an exception is detected.

[0123] As can be seen from the above, the embodiment of the present application carries out verification and analysis on the received data set preprocessing configuration file, and replaces variable parameters in the file with values in the runtime context to generate a directed graph representing the dependency relationship of the preprocessing steps. Then, the corresponding target main function is called according to the configuration file, the target plug-in function registered by the preset decorator in the preset plug-in algorithm pool is loaded through the target main function, the target machine learning data set corresponding to the configuration file is read from the preset hierarchical storage directory, the target machine learning data set is parallelly preprocessed according to the dependency relationship directed graph with the help of the thread pool, and the preprocessing result is saved to the preset hierarchical storage directory. At the same time, the execution state of each step is recorded in the preprocessing process to support breakpoint continuation, the processing progress of each file data is continuously tracked and displayed to the system front end, and the preset strategy is processed when an exception is monitored. In this way, through the above process of the embodiment of the present application, the verification and analysis of the configuration file and the variable replacement ensure the accuracy and adaptability of the preprocessing rules, the dependency relationship directed graph is generated, the logic between the steps is sorted out, and a reliable basis is provided for parallel processing. The calling of the corresponding main function and plug-in function enhances the flexibility and expansibility of the preprocessing, and can adapt to the processing needs of different types of data. The thread pool parallel processing combined with the read-write management of the hierarchical storage directory improves the preprocessing efficiency and avoids data storage chaos. The execution state recording, progress tracking and exception handling mechanism not only guarantees the stability of the processing process, but also facilitates the continuous processing after interruption, enables the user to master the progress in real time and respond to problems in time, and can optimize the machine learning data set preprocessing method to improve the efficiency, stability and flexibility of the data set preprocessing.

[0124] In some specific embodiments, the configured machine learning data set preprocessing method can further include:

[0125] An algorithm pool scanning unit is configured to scan the preset plug-in algorithm pool after the data set preprocessing system is started to obtain function meta information of a plurality of plug-in functions in the preset plug-in algorithm pool.

[0126] A form generating unit is configured to count the function meta information to generate a corresponding plug-in function information list, and generate a function configuration form of the plurality of plug-in functions according to the plug-in function information list.

[0127] A form display unit is configured to display the function configuration form to the front end, so as to obtain the data set preprocessing configuration file through the front end. The data set preprocessing configuration file is a JSON file configured based on the function configuration form and a preset configuration file rule, and related to preprocessing of the target machine learning data set.

[0128] In some specific embodiments, the parameter replacement module 11 can specifically include:

[0129] a file checking unit configured to check a received dataset preprocessing configuration file based on a target file checking tool and the plugin function information list, and determine a preprocessing step list in the dataset preprocessing configuration file after the checking is passed;

[0130] a condition determining unit configured to analyze parameters of each preprocessing step in the preprocessing step list to determine a corresponding precondition branch, and determine whether there is an unconfigured parameter dependent step to obtain a corresponding determination result;

[0131] a parameter identifying unit configured to insert the parameter dependent step into the preprocessing step list to obtain an updated preprocessing step list, and identify a variable parameter in the updated preprocessing step list, if the determination result represents that there is the unconfigured parameter dependent step;

[0132] a parameter replacing unit configured to replace the variable parameter with a value in a runtime context to generate a dependency directed graph representing dataset preprocessing steps based on the obtained replaced preprocessing step list and the precondition branch.

[0133] In some embodiments, the function loading module 12 can specifically include:

[0134] a name determining unit configured to determine a data type configured in the dataset preprocessing configuration file and an algorithm function name corresponding to each preprocessing step;

[0135] a first function calling unit configured to call the text processing main function if the data type is a text data type, to load a target plugin function with the algorithm function name in a preset plugin algorithm pool through the text processing main function;

[0136] a second function calling unit configured to call the table processing main function if the data type is a table data type, to load a target plugin function with the algorithm function name in a preset plugin algorithm pool through the table processing main function;

[0137] a third function calling unit configured to call the image processing main function if the data type is an image data type, to load a target plugin function with the algorithm function name in a preset plugin algorithm pool through the image processing main function.

[0138] In some embodiments, the preset hierarchical storage directory is a hierarchical storage directory based on a target object storage service, including a first folder, a second folder, and a third folder; the first folder is used to store an original machine learning dataset to be preprocessed; the second folder is used to store intermediate processing results generated in a preprocessing process of the original machine learning dataset; and the third folder is used to store a data preprocessing result obtained after preprocessing the original machine learning dataset.

[0139] Correspondingly, the dataset preprocessing module 13 can specifically include:

[0140] A dataset reading unit is configured to determine a parallelism degree configured in the dataset preprocessing configuration file, and read a target machine learning dataset corresponding to the dataset preprocessing configuration file from the first folder of the preset hierarchical storage directory.

[0141] A parameter verification unit is configured to obtain a parameter verification model of the target plug-in function by using a parameter information obtaining function, and verify the variable parameter based on the parameter verification model to obtain a corresponding verification result.

[0142] A data preprocessing submodule is configured to, if the verification result indicates that the variable parameter passes the verification, perform parallel preprocessing on each file data in the target machine learning dataset by using a thread pool according to the processing task type of the dependency relationship directed graph and the parallelism degree, to obtain a corresponding preprocessing result.

[0143] A result saving unit is configured to save the preprocessing result into the third folder, and save intermediate processing results generated in the preprocessing process into the second folder.

[0144] Correspondingly, the data preprocessing submodule can specifically include:

[0145] A block reading unit is configured to, if a data type of the target machine learning dataset is a table data type, perform block reading on data in the target machine learning dataset, and perform parallel preprocessing on the target machine learning dataset in a streaming processing manner.

[0146] A detection edge unit is configured to, if a data type of the target machine learning dataset is an image data type, perform a caching operation on an image decoding result by using a target caching algorithm, and perform detection edge on a thumbnail of image data in the target machine learning dataset to determine a target key region of the image data, and perform full-resolution processing on the target key region.

[0147] In some embodiments, the state recording module 14 can specifically include:

[0148] The state code returning unit is configured to, if the variable parameter exception is monitored, intercept the variable parameter and return a corresponding target response state code.

[0149] The file preprocessing unit is configured to, if the current to-be-processed data file is monitored to be damaged, record the to-be-processed data file and perform parallel preprocessing on other unprocessed data files in the target machine learning data set except the to-be-processed data file.

[0150] The event feedback unit is configured to, if the currently executed plug-in algorithm is monitored to crash, perform retry execution on the abnormal plug-in algorithm, record a corresponding retry number, and if the retry number reaches a preset retry number threshold, feed back the current abnormal event.

[0151] The module installation unit is configured to, if the software library or module relied on by the currently executed plug-in algorithm is monitored to be missing, install the missing software library or module.

[0152] In some specific embodiments, the configured machine learning data set preprocessing apparatus can further include:

[0153] The information determining unit is configured to define a parameter verification model of the new algorithm function by using a target software library and determine function meta information of the new algorithm function.

[0154] The decorator adding unit is configured to add the preset decorator in the code of the new algorithm function to obtain an updated new algorithm function.

[0155] The function adding unit is configured to add the updated new algorithm function to the preset plug-in algorithm pool based on the function meta information and the parameter verification model, so that the new algorithm function is registered based on the preset decorator when the data set preprocessing system is started.

[0156] Further, the embodiment of the application further discloses an electronic device, Figure 4 is a structure diagram of an electronic device 20 according to an exemplary embodiment, and the content in the figure cannot be considered as any limitation on the use range of the application. The electronic device 20 can specifically include at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25 and a communication bus 26. The memory 22 is used to store a computer program, the computer program is loaded and executed by the processor 21, and the related steps in the configured machine learning data set preprocessing method disclosed in any of the preceding embodiments are implemented. In addition, the electronic device 20 in the embodiment can be an electronic computer.

[0157] In this embodiment, the power supply 23 is configured to provide operating voltage for each hardware device on the electronic device 20; the communication interface 24 is configured to create a data transmission channel between the electronic device 20 and external devices, and the communication protocol followed by the communication interface 24 can be any communication protocol applicable to the technical solution of the present application, which is not limited specifically herein; the input / output interface 25 is configured to obtain external input data or output data to the outside, and the specific interface type can be selected according to the specific application needs, which is not limited specifically herein.

[0158] In addition, the memory 22 as a carrier of resource storage can be a read-only memory, a random access memory, a magnetic disk or an optical disk, etc., and the resources stored thereon can include an operating system 221, a computer program 222, etc., and the storage mode can be temporary storage or permanent storage.

[0159] The operating system 221 is configured to manage and control each hardware device and the computer program 222 on the electronic device 20, and can be Windows Server, Netware, Unix, Linux, etc. In addition to the computer program capable of completing the configured machine learning data set preprocessing method executed by the electronic device 20 disclosed in any of the foregoing embodiments, the computer program 222 can further include a computer program capable of completing other specific work.

[0160] Further, the present application also discloses a computer readable storage medium for storing a computer program; wherein the computer program is executed by a processor to implement the configured machine learning data set preprocessing method disclosed above. For the specific steps of the method, please refer to the corresponding content disclosed in the foregoing embodiments, which will not be repeated here.

[0161] Each embodiment in the specification is described in a progressive manner, and each embodiment focuses on the difference from other embodiments. For the same or similar parts between each embodiment, please refer to each other. For the device disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple, and the relevant parts are described in the method part.

[0162] Those skilled in the art will further appreciate that the units and algorithm steps of the various examples described in connection with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or combinations of both. To clearly illustrate this interchangeability of hardware and software, various examples were described above generally in terms of their functionality, without referring to the corresponding

[0163] The steps of a method or algorithm described in connection with the embodiments disclosed herein can be embodied directly in hardware, in a software module executed by a processor, or in a combination of the two. A software module can reside in random access memory (RAM), flash memory, read-only memory (ROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), registers, hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art. An exemplary storage medium is tangible.

[0164] Finally, it should be noted that the terms "comprises", "comprising", or other variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but can also include other elements not expressly listed or inherent to such process, method, article, or apparatus. An element proceeded by "comprises... a" does not, without more constraints, foreclose the existence of additional identical elements in the process, method, article, or apparatus that comprises the element.

[0165] The above detailed description has set forth various examples of the technology disclosed herein. The description is purposefully rendered in this form for the purpose of providing clear and comprehensive disclosure of the technology disclosed herein, and, thus, no additional limitations or scope should be inferred therefrom for purposes of appropriate appreciation of the technology disclosed herein.

Claims

1. A configurable machine learning dataset preprocessing method, characterized in that, Applications in dataset preprocessing systems include: The received dataset preprocessing configuration file is validated and parsed, and the variable parameters in the dataset preprocessing configuration file are replaced with values ​​from the runtime context to generate a directed graph representing the dependency relationships of the dataset preprocessing steps. The target main function is called based on the dataset preprocessing configuration file to load the target plugin function from the preset plugin algorithm pool; the target main function is any one of text processing main function, table processing main function, and image processing main function; the target plugin function is an algorithm function that is registered based on a preset decorator and added to the preset plugin algorithm pool. The target machine learning dataset corresponding to the dataset preprocessing configuration file is read from the preset hierarchical storage directory, and the target machine learning dataset is preprocessed in parallel according to the directed graph of the dependency relationship through a thread pool, and the preprocessing results are saved to the preset hierarchical storage directory. During the preprocessing process, the execution status of each preprocessing step is recorded so that breakpoint resume can be performed based on the execution status. The processing progress of each file in the target machine learning dataset is continuously tracked so that the tracking results can be displayed on the front end of the dataset preprocessing system. When an anomaly is detected, it is handled according to the preset anomaly handling strategy.

2. The configurable machine learning dataset preprocessing method according to claim 1, characterized in that, Before verifying and parsing the received dataset preprocessing configuration file, the process also includes: After the dataset preprocessing system is started, the preset plugin algorithm pool is scanned to obtain the function meta-information of several plugin functions in the preset plugin algorithm pool; The function metadata is statistically analyzed to generate a corresponding list of plugin function information, and a function configuration form for the several plugin functions is generated based on the list of plugin function information. The function configuration form is displayed to the front end so that the dataset preprocessing configuration file can be obtained through the front end; the dataset preprocessing configuration file is a JSON file that configures the preprocessing-related information of the target machine learning dataset based on the function configuration form and preset configuration file rules.

3. The configurable machine learning dataset preprocessing method according to claim 2, characterized in that, The process of validating and parsing the received dataset preprocessing configuration file, and replacing the variable parameters in the dataset preprocessing configuration file with values ​​from the runtime context to generate a directed graph representing the dependency relationships of the dataset preprocessing steps, includes: The received dataset preprocessing configuration file is verified based on the target file verification tool and the plugin function information list, and after the verification is passed, the list of preprocessing steps in the dataset preprocessing configuration file is determined. The parameters of each preprocessing step in the preprocessing step list are analyzed to determine the corresponding precondition branches and to determine whether there are any unconfigured parameter-dependent steps, so as to obtain the corresponding judgment results. If the judgment result indicates that there is an unconfigured parameter dependency step, then the parameter dependency step is inserted into the preprocessing step list to obtain the corresponding updated preprocessing step list, and the variable parameters in the updated preprocessing step list are identified. The variable parameters are replaced with values ​​from the runtime context to generate a directed graph representing the dependencies of the dataset preprocessing steps based on the obtained list of replaced preprocessing steps and the precondition branches.

4. The configurable machine learning dataset preprocessing method according to claim 1, characterized in that, The step of calling the target main function based on the dataset preprocessing configuration file, and loading the target plugin function from the preset plugin algorithm pool through the target main function, includes: Determine the data types configured in the dataset preprocessing configuration file and the names of the algorithm functions corresponding to each preprocessing step; If the data type is a text data type, then the text processing main function is called to load the target plugin function named the algorithm function from the preset plugin algorithm pool through the text processing main function; If the data type is a table data type, then the table processing main function is called to load the target plugin function named the algorithm function from the preset plugin algorithm pool through the table processing main function; If the data type is an image data type, then the image processing main function is called to load the target plugin function named the algorithm function from the preset plugin algorithm pool.

5. The configurable machine learning dataset preprocessing method according to claim 1, characterized in that, The preset hierarchical storage directory is a hierarchical storage directory based on the target object storage service, including a first folder, a second folder, and a third folder; the first folder is used to store the original machine learning dataset to be preprocessed; The second folder is used to store intermediate processing results generated during the preprocessing of the original machine learning dataset; The third folder is used to store the data preprocessing results obtained after preprocessing the original machine learning dataset; Accordingly, the step of reading the target machine learning dataset corresponding to the dataset preprocessing configuration file from the preset hierarchical storage directory, performing parallel preprocessing of the target machine learning dataset according to the directed graph of the dependency relationship using a thread pool, and saving the preprocessing results to the preset hierarchical storage directory includes: Determine the degree of parallelism configured in the dataset preprocessing configuration file, and read the target machine learning dataset corresponding to the dataset preprocessing configuration file from the first folder of the preset hierarchical storage directory; The parameter verification model of the target plugin function is obtained using the parameter information acquisition function, and the variable parameters are verified based on the parameter verification model to obtain the corresponding verification results. If the verification result indicates that the variable parameter passes the verification, then according to the processing task type of the directed graph of the dependency relationship and the degree of parallelism, the data of each file in the target machine learning dataset is preprocessed in parallel through a thread pool to obtain the corresponding preprocessing result; The preprocessing results are saved to the third folder, and the intermediate processing results generated during the preprocessing process are saved to the second folder. Accordingly, the parallel preprocessing of each file in the target machine learning dataset includes: If the data type of the target machine learning dataset is a tabular data type, then the data in the target machine learning dataset is read in blocks and the target machine learning dataset is preprocessed in parallel in a streaming manner. If the data type of the target machine learning dataset is an image data type, then the image decoding result is cached using a target caching algorithm, and edges are detected on the thumbnails of the image data in the target machine learning dataset to determine the target key region of the image data, and the target key region is processed at full resolution.

6. The configurable machine learning dataset preprocessing method according to claim 1, characterized in that, The step of handling an anomaly according to a preset anomaly handling strategy when an anomaly is detected includes: If an abnormality is detected in the variable parameter, the variable parameter will be intercepted and the corresponding target response status code will be returned. If the current data file to be processed is detected to be corrupted, the data file to be processed is recorded, and parallel preprocessing is performed on the other unprocessed data files in the target machine learning dataset, excluding the data file to be processed. If a crash is detected in the currently executing plugin algorithm, the abnormal plugin algorithm will be retried and the corresponding number of retries will be recorded. If the number of retries reaches a preset retry threshold, feedback will be given for the current abnormal event. If it is detected that the software library or module on which the currently executed plug-in algorithm depends is missing, then the missing software library or module will be installed.

7. The configurable machine learning dataset preprocessing method according to claim 1, characterized in that, Also includes: The parameter verification model of the newly added algorithm function is defined using the target software library, and the function meta-information of the newly added algorithm function is determined; Add the preset decorator to the code of the newly added algorithm function to obtain the updated newly added algorithm function; Based on the function meta-information and the parameter verification model, the updated new algorithm function is added to the preset plugin algorithm pool so that when the dataset preprocessing system starts, the new algorithm function is registered based on the preset decorator.

8. A configurable machine learning dataset preprocessing apparatus, characterized in that, Applications in dataset preprocessing systems include: The parameter replacement module is used to verify and parse the received dataset preprocessing configuration file, and replace the variable parameters in the dataset preprocessing configuration file with values ​​from the runtime context to generate a directed graph representing the dependency relationships of the dataset preprocessing steps. The function loading module is used to call the target main function based on the dataset preprocessing configuration file, so as to load the target plugin function in the preset plugin algorithm pool through the target main function; the target main function is any one of the text processing main function, table processing main function, and image processing main function; the target plugin function is an algorithm function that is registered based on a preset decorator and added to the preset plugin algorithm pool. The dataset preprocessing module is used to read the target machine learning dataset corresponding to the dataset preprocessing configuration file from the preset hierarchical storage directory, and to perform parallel preprocessing of the target machine learning dataset according to the directed graph of the dependency relationship through a thread pool, and to save the preprocessing results to the preset hierarchical storage directory. The status recording module is used to record the execution status of each preprocessing step during the preprocessing process, so as to resume interrupted transmission based on the execution status, continuously track the processing progress of each file data in the target machine learning dataset, display the tracking results to the front end of the dataset preprocessing system, and handle the detected anomalies according to the preset anomaly handling strategy.

9. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor for executing the computer program to implement the configurable machine learning dataset preprocessing method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, Used to store computer programs; wherein, when the computer programs are executed by a processor, they implement the configurable machine learning dataset preprocessing method as described in any one of claims 1 to 7.

Images