End-to-end containerized big data model construction method, device, equipment and medium

Abstract

The present application discloses an end-to-end containerized big data analysis model construction method, device, equipment and storage medium, which relates to the field of machine learning, including: obtaining raw data for classification to obtain structured data and unstructured data, and grouping to obtain a target data set; determining the parameter information of the initial model, generating an algorithm configuration file based on the parameter information, and inputting it into a preset virtualized container to verify the mapping relationship between the target data set and the algorithm configuration file; if correct, using a preset executor to train the initial model according to the mapping relationship, and using a preset executor to call a preset parameter scheduler during the training process to dynamically adjust the hyperparameters of a preset optimizer to optimize the initial model and generate a target model. Through data set classification, algorithm adaptation and executors, the flexibility, efficiency and scalability of model construction in different scenarios are achieved, and the control of virtualized containers achieves environmental isolation to ensure the completion of training tasks.

Description

Technical Field

[0001] The present invention relates to the field of machine learning, and in particular to an end-to-end containerized big data analysis model construction method, device, equipment and storage medium. Background Art

[0002] In today's era of information explosion, big data analysis and artificial intelligence have become important forces driving social development and business innovation. However, building efficient and accurate AI (Artificial Intelligence) models often faces a series of challenges and problems. These problems include model building requirements in different application scenarios, the diversity and complexity of data sets, algorithm adaptation, and the complexity of the training process.

[0003] First, the requirements for AI models in different application scenarios vary greatly, and a flexible model building platform is needed to meet the diverse needs. Traditional model building methods often require writing specific code for different scenarios, which increases the complexity of development and maintenance. Secondly, the diversity and complexity of data sets are also a major challenge in building AI models. Data in the real world includes both structured data (such as tabular data) and unstructured data (such as images, text, etc.). Different data types require different preprocessing, feature extraction and annotation methods, so a unified data set implementation method is needed. In addition, algorithm adaptation and training process are also key issues in building AI models. Different algorithms have different requirements for hyperparameters, training strategies, and optimizers, making the model building process complex and error-prone. Moreover, the conventional training process often requires writing a large amount of training code and processes, which is not convenient for rapid experimentation and iteration. Therefore, how to provide a flexible, efficient, and unified AI model building platform to meet the model building needs in different application scenarios is an urgent problem to be solved in this field. Summary of the invention

[0004] In view of this, the purpose of the present invention is to provide an end-to-end containerized big data analysis model construction method, device, equipment and storage medium, which can achieve the flexibility, efficiency and scalability of model construction in different scenarios through data set classification, algorithm adaptation and executor, and realize environmental isolation by controlling the virtualized container to ensure the completion of the training task. The specific scheme is as follows:

[0005] In a first aspect, the present application provides an end-to-end containerized big data analysis AI model construction method, including:

[0006] Acquire original data sent by a preset data source, classify the original data to obtain corresponding structured data and unstructured data, generate an initial data set based on the structured data and the unstructured data, and group the initial data set according to a preset grouping rule to obtain a target data set;

[0007] Determine parameter information of the initial model, generate an algorithm configuration file in a preset format according to the parameter information, input the algorithm configuration file into a preset virtualization container, and use the preset virtualization container to verify the mapping relationship between the target data set and the algorithm configuration file;

[0008] If the mapping relationship is correct, the preset executor is used to train the initial model according to the mapping relationship, and during the training process, the preset executor is used to call the preset parameter scheduler to dynamically adjust the hyperparameters of the preset optimizer, so as to use the preset optimizer to optimize the initial model to generate the target model.

[0009] Optionally, generating an initial data set based on the structured data and the unstructured data includes:

[0010] Performing data quality checks, data cleaning and conversion operations on the fields of the structured data, and generating corresponding structured data sets;

[0011] Annotating the unstructured data to generate an unstructured data set in a label format;

[0012] An initial data set is generated according to the structured data set and the unstructured data set.

[0013] Optionally, after the initial data set is grouped according to a preset grouping rule to obtain a target data set, the method further includes:

[0014] The target data set is input into the preset virtualization container, and an application programming interface corresponding to the target data set is provided in the form of a network request.

[0015] Optionally, generating an algorithm configuration file in a preset format according to the parameter information includes:

[0016] The parameter information is assembled into a parameter dictionary according to a preset algorithm, and the parameter dictionary is saved as an algorithm configuration file in .yml or .json format.

[0017] Optionally, the using a preset executor to train the initial model according to the mapping relationship includes:

[0018] The parameter dictionary in the algorithm configuration file is read, and the corresponding initial model is loaded according to the parameter dictionary using the preset virtualization container, so that the corresponding target data set is read according to the mapping relationship using the preset executor, and the initial model is trained.

[0019] Optionally, the verifying the mapping relationship between the target data set and the algorithm configuration file by using the preset virtualization container includes:

[0020] The preset virtualization container is used to determine whether the features corresponding to the target data set and the algorithm configuration file match. If they match, it is determined that the mapping relationship between the target data set and the algorithm configuration file is correct.

[0021] Optionally, the using a preset executor to train the initial model according to the mapping relationship further includes:

[0022] The preset executor is used to set a hook so as to obtain the user's customized request during the training process according to the hook, and train the initial model through the customized request.

[0023] In a second aspect, the present application provides an end-to-end containerized big data analysis model construction device, including:

[0024] A data set grouping module, which obtains raw data sent by a preset data source, classifies the raw data to obtain corresponding structured data and unstructured data, generates an initial data set based on the structured data and the unstructured data, and groups the initial data set according to a preset grouping rule to obtain a target data set;

[0025] A configuration file verification module, used to determine parameter information of the initial model, generate an algorithm configuration file in a preset format according to the parameter information, input the algorithm configuration file into a preset virtualization container, and use the preset virtualization container to verify the mapping relationship between the target data set and the algorithm configuration file;

[0026] A model training module is used to train the initial model according to the mapping relationship using a preset executor if the mapping relationship is correct, and to call a preset parameter scheduler using the preset executor to dynamically adjust the hyperparameters of a preset optimizer during the training process, so as to optimize the initial model using the preset optimizer to generate a target model.

[0027] In a third aspect, the present application provides an electronic device, comprising a processor and a memory; wherein the memory is used to store a computer program, and the computer program is loaded and executed by the processor to implement the aforementioned end-to-end containerized big data analysis model construction method.

[0028] In a fourth aspect, the present application provides a computer-readable storage medium for storing a computer program, which, when executed by a processor, implements the aforementioned end-to-end containerized big data analysis model construction method.

[0029] In the present application, raw data sent by a preset data source is obtained, the raw data is classified to obtain corresponding structured data and unstructured data, an initial data set is generated based on the structured data and unstructured data, and the initial data set is grouped according to preset grouping rules to obtain a target data set; parameter information of the initial model is determined, an algorithm configuration file in a preset format is generated according to the parameter information, and the algorithm configuration file is input into a preset virtualization container, and the mapping relationship between the target data set and the algorithm configuration file is verified using the preset virtualization container; if the mapping relationship is correct, the initial model is trained according to the mapping relationship using a preset executor, and during the training process, the preset parameter scheduler is called by the preset executor to dynamically adjust the hyperparameters of the preset optimizer, so as to optimize the initial model using the preset optimizer to generate a target model. In this way, the present application can conveniently process different types of data and form a unified data set format through a unified data set implementation method, and through the implementation method of the adaptation algorithm, different types of algorithms can be built and trained through one-click training. At the same time, the introduction of the executor as the core module of the training engine can realize efficient execution and management of training, testing and reasoning tasks, avoiding multiple independent steps such as data collection, data preprocessing, feature extraction, model selection, hyperparameter tuning, etc. required when building AI models, simplifying the complexity of building AI models for non-professionals. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0031] Figure 1 A flow chart of an end-to-end containerized big data analysis model construction method provided for this application;

[0032] Figure 2 A data set integration flow chart provided for this application;

[0033] Figure 3 A model training flow chart provided for this application;

[0034] Figure 4 A flow chart of a specific end-to-end containerized big data analysis model construction method provided for this application;

[0035] Figure 5 A containerized flow chart provided for this application;

[0036] Figure 6 A deep learning flowchart provided for this application;

[0037] Figure 7 A schematic diagram of the structure of an end-to-end containerized big data analysis model building device provided in this application;

[0038] Figure 8 A structural diagram of an electronic device provided for this application. DETAILED DESCRIPTION

[0039] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0040] At present, building efficient and accurate AI models often faces problems such as different model building requirements in different application scenarios, the diversity and complexity of data sets, algorithm adaptation and the complexity of the training process. This application can easily process different types of data and form a unified data set format through a unified data set implementation method, and through the implementation method of the adaptation algorithm, different types of algorithms can be built and trained through one-click training. At the same time, the introduction of the executor as the core module of the training engine can realize the efficient execution and management of training, testing and reasoning tasks.

[0041] See also Figure 1 As shown, the embodiment of the present invention discloses an end-to-end containerized big data analysis model construction method, including:

[0042] Step S11, obtaining original data sent by a preset data source, classifying the original data to obtain corresponding structured data and unstructured data, generating an initial data set based on the structured data and the unstructured data, and grouping the initial data set according to preset grouping rules to obtain a target data set.

[0043] In this embodiment, the original data in the data source is first obtained, the original data is classified to obtain the corresponding structured data and unstructured data, and an initial data set is generated based on the classified structured data and unstructured data, and then the initial data set is grouped according to the preset grouping rules to obtain the target data set. First, the original data in the data source is cleaned, extracted, filtered, and annotated to form a data set, and the data set is grouped to form a training set and a test set, so that the adaptation algorithm can be assembled for different data sets in the future, and then the training and evaluation processes are carried out in the container.

[0044] In the process of classifying data sources and further processing and labeling, we first perform data quality checks, data cleaning and conversion operations on the fields of structured data, and generate corresponding structured data sets; and label unstructured data to generate unstructured data sets in label format; then generate the initial data set based on the structured data set and the unstructured data set. Figure 2 As shown in the figure, the generation of the dataset specifically includes the following steps:

[0045] Data source classification: classify raw data into structured data and unstructured data according to its characteristics. The above-mentioned structured data refers to data with fixed formats and fields, such as database tables, CSV (Comma-Separated Values) files, etc. Unstructured data refers to data without fixed formats or fields, such as text documents, images, audio, video, etc.; Structured data analysis: perform field-level analysis on structured data. First, perform data quality checks, data cleaning and conversion operations on each field to ensure data accuracy and consistency. Then, according to analysis requirements, perform statistics, aggregation, sorting and other operations on the fields to obtain useful information and insights; Unstructured data annotation: Annotate unstructured data to generate a data set in a labeled format. Annotation refers to assigning specific labels or tags to unstructured data for subsequent model training and analysis, such as , image data can be annotated by object detection, image classification, etc., and text data can be annotated by named entity recognition, sentiment analysis, etc.; Dataset grouping and sampling: According to actual needs, the data set can be grouped and sampled. Grouping can divide the data set into different subsets according to certain features or attributes so that it can be more accurately applied to different model building tasks. Sampling can select some samples from the overall data set for rapid verification and debugging during the model building process; Dataset format is unified: After the previous processing steps, both structured data and unstructured data have been processed and annotated, and a unified data set format has been generated. These data sets can be stored in a specific file format. By classifying data sources into structured data and unstructured data types, structured data is analyzed at the field level, and unstructured data is annotated to form a data set in a label format. The data set is further grouped, sampled, and other operations are performed to finally form a unified type of data set, which is obtained through network requests through containers.

[0046] It should be pointed out that after the initial data set is grouped according to the preset grouping rules to obtain the target data set, the target data set is input into the preset virtualization container, and the application programming interface corresponding to the target data set is provided in the form of a network request. By sending the final data set into the container and providing it to the user in the form of a network request, the user can obtain the data set, including the original data and the data set label, by accessing the API (Application Programming Interface) interface in the container for subsequent model building and analysis. Through the above data set implementation method, this embodiment provides a unified, flexible and efficient method that enables users to easily process and use structured data and unstructured data for model building and analysis. In the training process of the AI ​​model, the storage requirements of massive data and the occupation of multi-task training files are two common problems. This embodiment can form different data sets for different data sources to quickly build a unique model, which solves the problem that the traditional method requires a large amount of storage space to save data, and each task requires an independent training file, resulting in a waste of storage resources and low training efficiency. At the same time, it also has good support for very large data sets, solving the storage problem of massive data and the technical problem of multi-task training file occupation.

[0047] Step S12, determining parameter information of the initial model, generating an algorithm configuration file in a preset format according to the parameter information, inputting the algorithm configuration file into a preset virtualization container, and using the preset virtualization container to verify the mapping relationship between the target data set and the algorithm configuration file.

[0048] In this embodiment, after determining the parameter information of the initial model, an algorithm configuration file in a preset format is generated according to the parameter information, and the algorithm configuration file is input into a preset virtualization container. The preset virtualization container is used to determine whether the features corresponding to the target data set and the algorithm configuration file match. If they match, it is determined that the mapping relationship between the target data set and the algorithm configuration file is correct.

[0049] Step S13: If the mapping relationship is correct, the initial model is trained according to the mapping relationship using the preset executor, and during the training process, the preset executor is used to call the preset parameter scheduler to dynamically adjust the hyperparameters of the preset optimizer, so as to use the preset optimizer to optimize the initial model and generate the target model.

[0050] In this embodiment, if the mapping relationship is determined to be correct, the preset executor is used to train the initial model according to the mapping relationship, and the preset executor is used to call the preset parameter scheduler to dynamically adjust the hyperparameters of the preset optimizer during the training process, so as to optimize the initial model using the preset optimizer to generate the target model. The above-mentioned executor (Runner) is responsible for executing training, testing and reasoning tasks, and managing the required components. The executor allows users to expand and execute custom logic by setting hooks (Hooks). The optimizer encapsulates and executes the back-propagation optimization model, and the parameter scheduler dynamically adjusts the optimizer hyperparameters.

[0051] In this application, if Figure 3 As shown, the original data sent by the preset data source is obtained, the original data is classified to obtain the corresponding structured data and unstructured data and an initial data set is generated, the target data set is generated through data annotation, normalization processing and group sampling, the parameter information of the initial model is determined, and the algorithm configuration file in a preset format is generated according to the parameter information, and the algorithm configuration file is input into the preset virtualization container, and the mapping relationship between the target data set and the algorithm configuration file is verified using the preset virtualization container. If the mapping relationship is correct, the preset executor is used to train the initial model according to the mapping relationship, and the preset executor is used to call the preset parameter scheduler during the training process to dynamically adjust the hyperparameters of the preset optimizer, so as to optimize the initial model using the preset optimizer to generate the target model. An end-to-end containerized big data analysis and AI model construction technology is provided to solve the above problems. In this way, the present application can easily process different types of data and form a unified data set format through a unified data set implementation method; different types of algorithms can be constructed and trained through one-click training through the implementation method of the adaptation algorithm; the introduction of the executor as the core module of the training engine can realize the efficient execution and management of training, testing and reasoning tasks.

[0052] Based on the previous embodiment, it can be seen that the present application can realize the execution and management of model training tasks by processing data sets, implementing adaptation algorithms, and introducing executors. Next, the process of introducing executors to realize model training tasks will be described in detail in this embodiment. Figure 4 As shown, the embodiment of the present application discloses a specific end-to-end containerized big data analysis model construction method, including:

[0053] Step S21, obtaining original data sent by a preset data source, classifying the original data to obtain corresponding structured data and unstructured data, generating an initial data set based on the structured data and the unstructured data, and grouping the initial data set according to preset grouping rules to obtain a target data set.

[0054] Step S22, determine the parameter information of the initial model, assemble the parameter information into a parameter dictionary according to a preset algorithm, save the parameter dictionary as an algorithm configuration file in .yml or .json format, and input the algorithm configuration file into a preset virtualization container, and use the preset virtualization container to verify the mapping relationship between the target data set and the algorithm configuration file.

[0055] The adaptation algorithm in this embodiment is implemented by assembling information such as hyperparameters and training strategies into a dictionary form, and can be input into the container in the form of a .yml or .json file, thereby realizing a unified interface for training tasks. In this way, different types of algorithms can be implemented by one-click training, and there is a mapping relationship between the data set and the algorithm. When training a large model, different algorithms may have different evaluation requirements and evaluation indicators, and this application can allow multiple types of algorithms to share an evaluation function during the training process, and achieve unified evaluation of multiple types of algorithms by introducing a unified evaluation framework and flexible algorithm interface design, and provide accurate performance indicators and monitoring information to guide model training and tuning, avoiding the traditional method of designing and implementing evaluation functions for each algorithm separately, resulting in code duplication and maintenance difficulties, and further increasing the complexity and time-consuming problems of the evaluation process, solving the technical problem of unified model evaluation when training a large model. It can be understood that in this embodiment, a traditional machine learning model or a deep learning model can be selected as the model architecture.

[0056] like Figure 5The above hyperparameters, training strategies and other information are assembled in the form of dictionaries, and can be input into the container through .yml or .json files, and a unified interface is provided to realize one-click training of different types of algorithms. At the same time, the dataset and the algorithm are verified to ensure the correct mapping relationship to ensure the correct execution of the algorithm. The specific steps of the above model training are as follows: Parameter assembly: According to the requirements of the specific algorithm, the hyperparameters, training strategies and other related information are assembled into a dictionary. The above parameters include but are not limited to learning rate, batch size, network structure, loss function, etc. By assembling these parameters into the form of dictionaries, the algorithm can be easily configured and managed; Configuration file input: Save the parameter dictionary as a .yml or .json file and send it to the container through file input, which is conducive to flexible modification and adjustment of parameters without modifying the code, thereby improving the adaptability and configurability of the algorithm; Dataset and algorithm verification: Verify the dataset and the algorithm in the container to ensure that there is a correct mapping relationship between them. The detection of the mapping relationship can be achieved by checking whether the characteristics of the dataset match the characteristics required by the algorithm. If there is a mismatch or error, corresponding error prompts and suggestions can be provided. One-click training: By reading the parameter dictionary in the configuration file, the container can automatically load the corresponding algorithm model according to the requirements of different algorithms, and use the specified data set for training. The one-click training method allows users to quickly start training tasks without manually writing complex training codes and processes. Through the implementation of the above-mentioned adaptation algorithm, this embodiment provides a unified interface, so that different types of data sets can be easily mapped and trained with different algorithms. In this way, the combination of containerization technology and big data analysis algorithms provides users with a unified platform, enabling them to easily build and deploy AI models to solve various practical problems. Users can easily select and apply various complex algorithms, and adjust and optimize parameters as needed to achieve efficient model building and training processes.

[0057] Through the above technical solution, an efficient model format conversion algorithm is introduced, which can quickly convert a model from one format to another. The algorithm is based on the mapping relationship between model structure and parameters. By optimizing the calculation and data transmission methods, it reduces the calculation and storage overhead in the conversion process, improves the conversion speed and efficiency, and solves the problem of rapid conversion and generation of model formats.

[0058] Step S23: If the mapping relationship is correct, read the parameter dictionary in the algorithm configuration file, use the preset virtualization container to load the corresponding initial model according to the parameter dictionary, so as to use the preset executor to read the corresponding target data set according to the mapping relationship, and train the initial model, and use the preset executor to set a hook during the training process, so as to obtain the user's custom request during the training process according to the hook, and train the initial model through the custom request.

[0059] In this embodiment, Figure 6 As shown in the figure, the core module of the training engine used in the model training process and model evaluation implementation is the executor, which is responsible for executing and managing various components in training, testing, and reasoning tasks. The above executor allows users to expand, insert, and execute custom logic by setting hooks, and execute these logics at specific locations of training, testing, and reasoning tasks. Figure 6In the , the dataset is responsible for building the dataset required for training, testing, and inference tasks, and passing the data to the model. In actual use, the dataset is usually encapsulated by the data loader, which starts multiple subprocesses to load the data to improve data loading efficiency. The model accepts data as input and outputs loss (in training) or predictions (in testing and inference). In cases such as distributed training, the model is usually encapsulated by a model wrapper such as MMDistributedDataParallel. The optimizer is responsible for performing backpropagation to optimize the model and provides a unified interface that supports functions such as mixed precision training and gradient accumulation. The parameter scheduler is responsible for dynamically adjusting the optimizer's hyperparameters, such as learning rate, momentum, etc., during training to improve the training effect. In the training interval or test phase, the metrics and evaluator are responsible for evaluating the performance of the model. The evaluator evaluates the model's predictions based on the dataset, while the metrics calculate specific evaluation indicators, such as recall and accuracy. In this way, through the collaborative work of the executor and the above components, the training engine can efficiently perform training, testing and reasoning tasks, and provide flexible scalability, allowing users to customize and insert logic to meet different needs and application scenarios. During the training process, this embodiment can use GPU (graphics processing unit, graphics processor) and CPU (Central Processing Unit, central processing unit) to train the model, support most of the current traditional machine learning and deep learning algorithms, support multiple application scenarios, and monitor and control containerization, realize environmental isolation, and ensure the completion of training tasks.

[0060] For a more specific processing procedure of the above step S21, reference may be made to the corresponding contents disclosed in the above embodiments, which will not be described in detail here.

[0061] Through the above technical solution, this embodiment classifies the original data to obtain structured data and unstructured data and generates an initial data set, and groups the initial data set to obtain a target data set. Determine the parameter information of the initial model, assemble the parameter information into a parameter dictionary according to the preset algorithm, and save the parameter dictionary as an algorithm configuration file in .yml or .json format, and input the algorithm configuration file into the preset virtualization container to verify the mapping relationship between the target data set and the algorithm configuration file. If the mapping relationship is correct, read the parameter dictionary in the algorithm configuration file, use the preset virtualization container to load the corresponding initial model according to the parameter dictionary, so that the preset executor can be used to read the corresponding target data set according to the mapping relationship, and train the initial model, and use the preset executor to set a hook during the training process, so as to obtain the user's custom request during the training process according to the hook, and train the initial model through the custom request. This embodiment achieves flexibility, efficiency and scalability of model construction in different application scenarios through data set classification, algorithm adaptation and core modules of the training engine, and provides a flexible, efficient and unified AI model construction platform that can meet the model construction requirements in different application scenarios, simplify the model construction process, improve development efficiency and model accuracy, and through containerization and end-to-end design, it can easily integrate different components and tools to achieve rapid iteration and deployment, solve the complexity of AI model construction, and realize end-to-end model construction technology.

[0062] See also Figure 7 As shown, the embodiment of the present application also discloses an end-to-end containerized big data analysis model construction device, including:

[0063] The data set grouping module 11 obtains the original data sent by the preset data source, classifies the original data to obtain corresponding structured data and unstructured data, generates an initial data set based on the structured data and the unstructured data, and groups the initial data set according to a preset grouping rule to obtain a target data set;

[0064] A configuration file verification module 12 is used to determine parameter information of the initial model, generate an algorithm configuration file in a preset format according to the parameter information, input the algorithm configuration file into a preset virtualization container, and use the preset virtualization container to verify the mapping relationship between the target data set and the algorithm configuration file;

[0065] The model training module 13 is used to train the initial model according to the mapping relationship using the preset executor if the mapping relationship is correct, and to call the preset parameter scheduler by the preset executor to dynamically adjust the hyperparameters of the preset optimizer during the training process, so as to optimize the initial model using the preset optimizer to generate a target model.

[0066] In this embodiment, the original data sent by the preset data source is obtained, the original data is classified to obtain the corresponding structured data and unstructured data, and an initial data set is generated based on the structured data and unstructured data, and the initial data set is grouped according to the preset grouping rules to obtain the target data set; the parameter information of the initial model is determined, and an algorithm configuration file in a preset format is generated according to the parameter information, and the algorithm configuration file is input into a preset virtualization container, and the mapping relationship between the target data set and the algorithm configuration file is verified by using the preset virtualization container; if the mapping relationship is correct, the initial model is trained according to the mapping relationship by using the preset executor, and the preset parameter scheduler is called by the preset executor during the training process to dynamically adjust the hyperparameters of the preset optimizer, so as to optimize the initial model by using the preset optimizer to generate the target model. In this way, the present application can conveniently process different types of data and form a unified data set format through a unified data set implementation method, and through the implementation method of the adaptation algorithm, different types of algorithms can be constructed and trained through one-click training, and the executor is introduced as the core module of the training engine, which can realize efficient execution and management of training, testing and reasoning tasks.

[0067] In some specific embodiments, the data set grouping module 11 specifically includes:

[0068] A first data preprocessing unit, configured to perform data quality inspection, data cleaning and conversion operations on the fields of the structured data, and generate a corresponding structured data set;

[0069] A second data preprocessing unit, used to label the unstructured data and generate an unstructured data set in a label format;

[0070] The data set generating unit is used to generate an initial data set according to the structured data set and the unstructured data set.

[0071] In some specific embodiments, the data set grouping module 11 further includes:

[0072] The interface determination unit is used to input the target data set into the preset virtualization container and provide an application programming interface corresponding to the target data set in the form of a network request.

[0073] In some specific embodiments, the configuration file verification module 12 specifically includes:

[0074] A parameter dictionary assembly unit is used to assemble the parameter information into a parameter dictionary according to a preset algorithm, and save the parameter dictionary as an algorithm configuration file in .yml or .json format.

[0075] In some specific embodiments, the model training module 13 specifically includes:

[0076] A model training unit is used to read the parameter dictionary in the algorithm configuration file, use the preset virtualization container to load the corresponding initial model according to the parameter dictionary, so as to use the preset executor to read the corresponding target data set according to the mapping relationship and train the initial model.

[0077] In some specific embodiments, the configuration file verification module 12 specifically includes:

[0078] The mapping judgment unit is used to determine whether the features corresponding to the target data set and the algorithm configuration file match by using the preset virtualization container, and if they match, determine that the mapping relationship between the target data set and the algorithm configuration file is correct.

[0079] In some specific embodiments, the model training module 13 further includes:

[0080] A hook setting unit is used to set a hook using the preset executor, so as to obtain a user's customized request during training according to the hook, and train the initial model through the customized request.

[0081] Furthermore, the present application also discloses an electronic device. Figure 8 This is a structural diagram of an electronic device 20 according to an exemplary embodiment. The content in the diagram cannot be regarded as any limitation on the scope of use of the present application.

[0082] Figure 8 A schematic diagram of the structure of an electronic device 20 provided in an embodiment of the present application. The electronic device 20 may specifically include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 is used to store a computer program, which is loaded and executed by the processor 21 to implement the relevant steps in the end-to-end containerized big data analysis AI model construction method disclosed in any of the aforementioned embodiments. In addition, the electronic device 20 in this embodiment may specifically be an electronic computer.

[0083] In this embodiment, the power supply 23 is used to provide working voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and the external device, and the communication protocol it follows is any communication protocol that can be applied to the technical solution of the present application, and is not specifically limited here; the input and output interface 25 is used to obtain external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs and is not specifically limited here.

[0084] In addition, the memory 22, as a carrier for storing resources, can be a read-only memory, a random access memory, a disk or an optical disk, etc. The resources stored thereon can include an operating system 221, a computer program 222, etc., and the storage method can be temporary storage or permanent storage.

[0085] Among them, the operating system 221 is used to manage and control the hardware devices and computer programs 222 on the electronic device 20, which can be Windows Server, Netware, Unix, Linux, etc. In addition to including a computer program that can be used to complete the end-to-end containerized big data analysis AI model construction method performed by the electronic device 20 disclosed in any of the aforementioned embodiments, the computer program 222 can further include a computer program that can be used to complete other specific tasks.

[0086] Furthermore, the present application also discloses a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, the aforementioned disclosed end-to-end containerized big data analysis AI model construction method is implemented. The specific steps of the method can be referred to the corresponding contents disclosed in the aforementioned embodiments, and will not be repeated here.

[0087] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0088] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0089] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0090] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0091] The technical solution provided by the present application is introduced in detail above. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea. At the same time, for general technicians in this field, according to the idea of ​​the present application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. An end-to-end containerized big data analysis model construction method, characterized in that: include: Acquire original data sent by a preset data source, classify the original data to obtain corresponding structured data and unstructured data, generate an initial data set based on the structured data and the unstructured data, and group the initial data set according to a preset grouping rule to obtain a target data set; Determine parameter information of the initial model, generate an algorithm configuration file in a preset format according to the parameter information, input the algorithm configuration file into a preset virtualization container, and use the preset virtualization container to verify the mapping relationship between the target data set and the algorithm configuration file; If the mapping relationship is correct, the initial model is trained according to the mapping relationship using the preset executor, and during the training process, the preset executor is used to call the preset parameter scheduler to dynamically adjust the hyperparameters of the preset optimizer, so as to optimize the initial model using the preset optimizer to generate a target model; The step of generating an algorithm configuration file in a preset format according to the parameter information includes: Assembling the parameter information into a parameter dictionary according to a preset algorithm, and saving the parameter dictionary as an algorithm configuration file in .yml or .json format; Accordingly, the using a preset executor to train the initial model according to the mapping relationship includes: Read the parameter dictionary in the algorithm configuration file, and use the preset virtualization container to load the corresponding initial model according to the parameter dictionary, so as to use the preset executor to read the corresponding target data set according to the mapping relationship, and train the initial model; Furthermore, the verifying the mapping relationship between the target data set and the algorithm configuration file by using the preset virtualization container includes: Determine whether the target data set and the features corresponding to the algorithm configuration file match using the preset virtualization container, and if so, determine that the mapping relationship between the target data set and the algorithm configuration file is correct; And, grouping the initial data set according to a preset grouping rule to obtain a target data set includes: The initial data set is grouped according to a preset grouping rule to obtain a plurality of target data sets, so as to construct the corresponding target model according to each target data set; the preset grouping rule is a rule constructed according to the characteristics or attributes of the data in the initial data set.

2. The end-to-end containerized big data analysis model construction method according to claim 1 is characterized in that: The generating an initial data set based on the structured data and the unstructured data includes: Performing data quality checks, data cleaning and conversion operations on the fields of the structured data, and generating corresponding structured data sets; Annotating the unstructured data to generate an unstructured data set in a label format; An initial data set is generated according to the structured data set and the unstructured data set.

3. The end-to-end containerized big data analysis model construction method according to claim 1 is characterized in that: After the initial data set is grouped according to the preset grouping rule to obtain the target data set, the method further includes: The target data set is input into the preset virtualization container, and an application programming interface corresponding to the target data set is provided in the form of a network request.

4. The end-to-end containerized big data analysis model construction method according to any one of claims 1 to 3, characterized in that: The using a preset actuator to train the initial model according to the mapping relationship also includes: The preset executor is used to set a hook so as to obtain the user's customized request during the training process according to the hook, and train the initial model through the customized request.

5. An end-to-end containerized big data analysis model construction device, characterized in that: include: A data set grouping module, which obtains raw data sent by a preset data source, classifies the raw data to obtain corresponding structured data and unstructured data, generates an initial data set based on the structured data and the unstructured data, and groups the initial data set according to a preset grouping rule to obtain a target data set; A configuration file verification module, used to determine parameter information of the initial model, generate an algorithm configuration file in a preset format according to the parameter information, input the algorithm configuration file into a preset virtualization container, and use the preset virtualization container to verify the mapping relationship between the target data set and the algorithm configuration file; A model training module, used for, if the mapping relationship is correct, using a preset executor to train the initial model according to the mapping relationship, and during the training process, using the preset executor to call a preset parameter scheduler to dynamically adjust the hyperparameters of a preset optimizer, so as to use the preset optimizer to optimize the initial model to generate a target model; The configuration file verification module specifically includes: A parameter dictionary assembly unit, used to assemble the parameter information into a parameter dictionary according to a preset algorithm, and save the parameter dictionary as an algorithm configuration file in .yml or .json format; Accordingly, the model training module specifically includes: A model training unit, configured to read the parameter dictionary in the algorithm configuration file, load the corresponding initial model according to the parameter dictionary using the preset virtualization container, so as to read the corresponding target data set according to the mapping relationship using the preset executor, and train the initial model; Furthermore, the configuration file verification module specifically includes: The mapping judgment unit is used to determine whether the features corresponding to the target data set and the algorithm configuration file match by using the preset virtualization container, and if they match, determine that the mapping relationship between the target data set and the algorithm configuration file is correct.

6. An electronic device, characterized in that: The electronic device includes a processor and a memory; wherein the memory is used to store a computer program, and the computer program is loaded and executed by the processor to implement the end-to-end containerized big data analysis model construction method as described in any one of claims 1 to 4.

7. A computer-readable storage medium, characterized in that: Used to store a computer program, which, when executed by a processor, implements the end-to-end containerized big data analysis model construction method as described in any one of claims 1 to 4.

Images