A modeling experiment system based on cloud native technology

By applying cloud-native technology in the modeling experimental system, building clusters and creating CRDs, writing controllers and Manager services, and using Webhook to process data and results, the problems of low efficiency and insufficient control capabilities in the existing technology are solved, and efficient and a stable modeling experimental environment.

CN114201857BActive Publication Date: 2025-01-07XIAMEN DEEPFUZHI TECH CO LTD
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202111390092.5
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-11-22
Publication Date
2025-01-07
Estimated Expiration
2041-11-22

AI Technical Summary

Technical Problem

The existing modeling experimental system is inefficient when N experiments are performed at one time, making it difficult to perform operations such as pause, recovery, stop and timeout abandonment, and it is difficult to provide a stable, efficient and controllable modeling experimental environment.

Method used

Based on cloud-native technology, build Kubernetes, Mysql and Ceph clusters, use Kbuilder to create CRDs, write experiments and model resource controllers, create Manager services to receive experimental requests, and inject download and collect containers through Webhooks to process data and results to achieve dynamic hypervisors. Parameter adjustment, provides an interface for experimental progress and result query, and supports pause, recovery and stop operations.

Benefits of technology

It improves the experimental efficiency and provides more modeling control capabilities, including pause, recovery and stop operations, sets the longest execution time of the experiment and the maximum number of failures, achieving a stable, efficient and controllable modeling experimental environment.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114201857B_ABST
    Figure CN114201857B_ABST
Patent Text Reader

Abstract

The present invention relates to the field of modeling technology of cloud native technology, and specifically discloses a modeling experiment system based on cloud native technology, including the following steps, S1: algorithm image; S2: data address; S3: algorithm hyperparameter; S4: modeling control configuration; S5: submitting the algorithm image, data address and modeling control configuration to the modeling experiment system; S6: the modeling experiment system provides an interface for querying the progress and results of the experiment, and pausing, resuming and stopping a single modeling in the experiment. The beneficial effects of the present invention are that by extending the functions of cloud native technology, only simple configuration is required to improve the efficiency of the experiment, complete the experiment faster, and have more control capabilities over the modeling in the experiment, including pausing, resuming and stopping, etc., and the longest execution time of an experiment and the maximum number of modeling failures can be set. In short, the system of the present invention can provide a stable, efficient and controllable environment for modeling experiments.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of modeling technology of cloud native technology, and in particular to a modeling experiment system based on cloud native technology. Background Art

[0002] Modeling refers to abstracting a specific problem into a certain type of problem and expressing it with a mathematical model. The purpose is to obtain a certain mathematical model that can solve a certain type of problem. The process is to obtain a mathematical model through algorithms, algorithm hyperparameters and data calculations. For example, an algorithm for identifying wearing a mask and a set of pictures of wearing masks can obtain a mathematical model through modeling. This model can be used to identify whether a picture is wearing a mask. An experiment is modeled N (>=1) times. Each modeling has the same algorithm, data and different algorithm hyperparameters. The mathematical model and indicators are output. After one modeling is completed, the next modeling begins, until N times. Finally, the optimal mathematical model is selected according to the indicators. Cloud Native is translated as cloud native. It is a concept proposed by Matt Stine. It is a collection of ideas, including DevOps, continuous delivery, microservices, agile infrastructure, Conway's law, etc., as well as the reorganization of companies according to business capabilities. Cloud Native includes both technology (microservices, agile infrastructure) and management (DevOps, continuous delivery, Conway's law, reorganization, etc.). Cloud Native can also be said to be a collection of a series of technologies and enterprise management methods.

[0003] However, in today's modeling experiment system, one experiment is modeled N times. When the N value is large, it will take a long time to complete the experiment, resulting in very low efficiency. It is also difficult to pause, resume, stop, or give up on a single modeling. In short, the efficiency is low, and it is difficult to perform more specific operations on a single modeling, and it is impossible to provide a stable, efficient, and controllable modeling experiment environment.

[0004] To solve the above problems, a modeling experiment system based on cloud native technology is proposed. Summary of the invention

[0005] The purpose of the present invention is to provide a modeling experiment system based on cloud native technology. First, a Kubernetes cluster, a Mysql cluster and a Ceph cluster are built; then, Kbuilder is used to quickly create CRDs of Kubernetes: experimental resources and modeling resources, and core function experimental resource controllers and modeling resource controllers are written. The experimental resource controller monitors the state changes of experimental resources and modeling resources and responds, and the modeling resource controller monitors the Kubernetes native Job resources and responds; secondly, a Manager service is created to receive modeling experiment requests; secondly, the experimental request is parameter verified and saved to Mysql; thirdly, a Scheduling service is created to obtain Mysql Experimental request, call Kubernetes to create experimental resources according to the experimental request; secondly, create Webhook, inject download container and collection container, which are used to download the data required by the algorithm container and collect the results of the algorithm container and report them to the Manager, so that the algorithm container can focus on modeling without having to deal with whether the data comes from Ceph or Hive, and where to report the results. Webhook can also modify the input parameters of the container in the Pod to dynamically change the algorithm hyperparameters; secondly, the Manager service receives feedback results from the experimental resource controller and the modeling resource controller, including the current status of the experiment, the current status of the modeling and the data after the experiment ends, and saves the results to Mysql and Ceph, which can solve the problems raised in the above background technology.

[0006] To achieve the above object, the present invention provides the following technical solution: a modeling experiment system based on cloud native technology, comprising the following steps:

[0007] S1: Algorithm image: Use Docker to generate an algorithm image from the algorithm, and then submit the algorithm image to the warehouse, which has an accessible algorithm image address;

[0008] S2: Data address: Use Ceph or Hive to save the data and have an accessible data address;

[0009] S3: Algorithm hyperparameters: initialized algorithm hyperparameters;

[0010] S4: Modeling control configuration: the number of modeling in the experiment N, the number of modeling that can be executed simultaneously, the maximum execution time of the experiment, the maximum time of a single modeling, the maximum number of modeling failures, etc.;

[0011] S5: Submit the algorithm image, data address and modeling control configuration to the modeling experiment system;

[0012] S6: The modeling experiment system provides an interface for querying the progress and results of the experiment, as well as pausing, resuming, and stopping a single model in the experiment.

[0013] Furthermore, the specific implementation steps of S1-S6 are as follows:

[0014] S101: Build a cluster;

[0015] S102: Use Kbuilder to quickly create Kubernetes CRDs: experimental resources and modeling resources, write core function experimental resource controllers and modeling resource controllers, the experimental resource controllers monitor the status changes of experimental resources and modeling resources and respond, and the modeling resource controllers monitor Kubernetes native Job resources and respond;

[0016] S103: Create a Manager service to receive modeling experiment requests;

[0017] S104: Verify the parameters of the experimental request and save them to MySQL;

[0018] S105: Create a Scheduling service to obtain the Mysql experiment request, and call Kubernetes to create experiment resources according to the experiment request;

[0019] S106: Create a Webhook and inject a download container and a collection container into it. These containers are used to download the data required by the algorithm container and collect the results of the algorithm container and report them to the Manager. This allows the algorithm container to focus on modeling without having to deal with whether the data comes from Ceph or Hive, or where to report the results. The Webhook can also modify the input parameters of the container in the Pod to dynamically change the algorithm hyperparameters.

[0020] S107: The Manager service receives feedback results from the experiment resource controller and the modeling resource controller.

[0021] Furthermore, the clusters built in S101 are respectively building a Kubernetes cluster, building a Mysql cluster, and building a Ceph cluster.

[0022] Furthermore, the experiment resource CRD in S102 has the following fields: experiment type, search space, optimizer configuration, total number of modeling, maximum experiment execution time, number of modeling executed in parallel, and maximum number of modeling failures.

[0023] Furthermore, the modeling resource CRD in S102 has the following fields: pause, resume, abandon, and whether to retain the Job resource after completion.

[0024] Furthermore, because experimental resources are created in S105, the experimental resource controller will respond at this time, create modeling resources according to the content of the experimental resources, and monitor the status changes of the modeling resources to respond. For example, if a total of 4 modeling resources are requested and 2 modeling resources are executed at the same time, the experimental resource controller will first create 2 modeling resources, and then create 2 more after waiting for completion.

[0025] Furthermore, in S105, because a modeling resource is created, the modeling resource controller will respond, create a Job resource according to the modeling resource content, and respond by monitoring the state change of the Job resource. Kubernetes creates a corresponding Pod resource according to the Job resource.

[0026] Furthermore, the Pod resource is internally composed of a storage module, a data screening module, a download container module, an algorithm container module and a collection container module, and the storage module, the data screening module, the download container module, the algorithm container module and the collection container module are connected to each other in sequence.

[0027] Furthermore, the storage module is used to store all the data transmitted by the Job controller, and then the data screening module screens out only the data required by the algorithm container, which is then downloaded by the download container module and used by the algorithm container module. The collection container module collects the results of the algorithm container and reports them to the Manager.

[0028] Furthermore, the feedback result in S107 includes the current state of the experiment, the current state of the model and the data after the experiment is completed, and the results are saved in Mysql and Ceph.

[0029] Compared with the prior art, the present invention has the following beneficial effects:

[0030] 1. The present invention proposes a modeling experiment system based on cloud native technology. By expanding the functions of cloud native technology, it only requires simple configuration to improve the efficiency of the experiment, complete the experiment faster, and have more control over the modeling in the experiment, including pause, resume and stop, etc. It can also set the maximum execution time of an experiment and the maximum number of modeling failures. In short, the system of the present invention can provide a stable, efficient and controllable environment for modeling experiments.

[0031] 2. The present invention proposes a modeling experiment system based on cloud native technology, which creates a Webhook and injects a download container and a collection container, which are respectively used to download the data required by the algorithm container and collect the results of the algorithm container and report them to the Manager; the Pod resource is composed of a storage module, a data screening module, a download container module, an algorithm container module and a collection container module. The storage module is used to store all the data transmitted by the Job controller, and then the data screening module screens out only the data required by the algorithm container, which is then downloaded by the download container module and used by the algorithm container module. The collection container module collects the results of the algorithm container and reports them to the Manager, allowing the algorithm container to focus on modeling without having to process whether the data comes from Ceph or Hive, and where to report the results. The Webhook can also modify the input parameters of the container in the Pod to dynamically change the algorithm hyperparameters. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] The drawings constituting a part of this application are used to provide a further understanding of this application, so that other features, purposes and advantages of this application become more obvious. The schematic embodiment drawings and their descriptions of this application are used to explain this application and do not constitute an improper limitation on this application. In the drawings:

[0033] Figure 1 This is a flowchart of the entire system flow of the modeling experimental system based on cloud native technology of the present invention;

[0034] Figure 2 This is a system workflow diagram of the modeling experiment system based on cloud native technology of the present invention;

[0035] Figure 3 It is a detailed system workflow diagram of the modeling experimental system based on cloud native technology of the present invention;

[0036] Figure 4 This is a schematic diagram of the internal modules of the Pod resources of the modeling experiment system based on cloud native technology of the present invention. DETAILED DESCRIPTION

[0037] In order to enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of this application.

[0038] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present application described here. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0039] In the present application, the terms "upper", "lower", "left", "right", "front", "back", "top", "bottom", "inner", "outer", "middle", "vertical", "horizontal", "lateral", "longitudinal" and the like indicate positions or positional relationships based on the positions or positional relationships shown in the drawings. These terms are mainly used to better describe the present application and its embodiments, and are not used to limit the indicated devices, elements or components to have a specific orientation, or to be constructed and operated in a specific orientation.

[0040] In addition, some of the above terms may be used to express other meanings in addition to indicating orientation or positional relationship. For example, the term "on" may also be used to express a certain dependency or connection relationship in some cases. For those of ordinary skill in the art, the specific meanings of these terms in this application can be understood according to specific circumstances.

[0041] In addition, the term "plurality" shall mean two or more.

[0042] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0043] Embodiment 1

[0044] See also Figure 1-2 , a modeling experiment system based on cloud native technology, including the following steps:

[0045] Step 1: Algorithm image: Use Docker to generate an algorithm image, and then submit the algorithm image to the warehouse, where the algorithm image address is accessible;

[0046] Step 2: Data address: Use Ceph or Hive to save the data and have an accessible data address;

[0047] Step 3: Algorithm hyperparameters: initialized algorithm hyperparameters;

[0048] Step 4: Modeling control configuration: the number of modeling in the experiment N, the number of modeling that can be executed simultaneously, the maximum execution time of the experiment, the maximum time for a single modeling, the maximum number of modeling failures, etc.;

[0049] Step 5: Submit the algorithm image, data address and modeling control configuration to the modeling experiment system;

[0050] Step 6: The modeling experiment system provides an interface for querying the progress and results of the experiment, as well as pausing, resuming and stopping individual modeling in the experiment.

[0051] By expanding the functions of cloud-native technology, only simple configuration is required to improve the efficiency of experiments, complete experiments faster, and have more control over modeling in experiments, including pause, resume, and stop. The maximum execution time of an experiment and the maximum number of modeling failures can also be set. In short, the system of the present invention can provide a stable, efficient, and controllable environment for modeling experiments.

[0052] See also Figure 1 and Figure 3 , a modeling experiment system based on cloud native technology. The specific implementation steps of the above system are as follows:

[0053] Step 1: Build clusters. The clusters built are Kubernetes cluster, MySQL cluster and Ceph cluster.

[0054] Step 2: Use Kbuilder to quickly create Kubernetes' CRD: experimental resources and modeling resources, write core function experimental resource controllers and modeling resource controllers, the experimental resource controllers monitor the status changes of experimental resources and modeling resources and respond, and the modeling resource controllers monitor Kubernetes native Job resources and respond;

[0055] The experiment resource CRD has the following fields: experiment type, search space, optimizer configuration, total number of modeling, maximum experiment execution time, number of modelings executed in parallel, and maximum number of modeling failures;

[0056] The modeling resource CRD has the following fields: pause, resume, abandon, and whether to retain the Job resource after completion;

[0057] Step 3: Create a Manager service to receive modeling experiment requests;

[0058] Step 4: Verify the parameters of the experimental request and save it to MySQL;

[0059] Step 5: Create a Scheduling service to obtain the Mysql experiment request, and call Kubernetes to create experimental resources according to the experiment request;

[0060] Because experimental resources are created, the experimental resource controller will respond at this time, create modeling resources according to the experimental resource content, and monitor the state changes of the modeling resources to respond. For example, if the request requires a total of 4 modeling resources and 2 modeling resources are executed at the same time, the experimental resource controller will first create 2 modeling resources and then create 2 more after waiting for completion;

[0061] Because modeling resources are created, the modeling resource controller will respond by creating Job resources based on the modeling resource content and monitoring the status changes of the Job resources to respond. Kubernetes will create corresponding Pod resources based on the Job resources.

[0062] Step 6: Create a Webhook and inject the download container and the collection container into it. These containers are used to download the data required by the algorithm container and collect the results of the algorithm container and report them to the Manager. This allows the algorithm container to focus on modeling without having to deal with whether the data comes from Ceph or Hive, or where to report the results. The Webhook can also modify the input parameters of the container in the Pod to dynamically change the algorithm hyperparameters.

[0063] In the seventh step, the Manager service receives feedback from the experiment resource controller and the modeling resource controller, including the current state of the experiment, the current state of the modeling, and the data after the experiment ends, and saves the results to Mysql and Ceph.

[0064] Embodiment 2

[0065] See also Figure 1-2 , a modeling experiment system based on cloud native technology, including the following steps:

[0066] Step 1: Algorithm image: Use Docker to generate an algorithm image, and then submit the algorithm image to the warehouse, where the algorithm image address is accessible;

[0067] Step 2: Data address: Use Ceph or Hive to save the data and have an accessible data address;

[0068] Step 3: Algorithm hyperparameters: initialized algorithm hyperparameters;

[0069] Step 4: Modeling control configuration: the number of modeling in the experiment N, the number of modeling that can be executed simultaneously, the maximum execution time of the experiment, the maximum time for a single modeling, the maximum number of modeling failures, etc.;

[0070] Step 5: Submit the algorithm image, data address and modeling control configuration to the modeling experiment system;

[0071] Step 6: The modeling experiment system provides an interface for querying the progress and results of the experiment, as well as pausing, resuming and stopping individual modeling in the experiment.

[0072] By expanding the functions of cloud-native technology, only simple configuration is required to improve the efficiency of experiments, complete experiments faster, and have more control over modeling in experiments, including pause, resume, and stop. The maximum execution time of an experiment and the maximum number of modeling failures can also be set. In short, the system of the present invention can provide a stable, efficient, and controllable environment for modeling experiments.

[0073] See also Figure 1 and Figure 3 , a modeling experiment system based on cloud native technology. The specific implementation steps of the above system are as follows:

[0074] Step 1: Build clusters. The clusters built are Kubernetes cluster, MySQL cluster and Ceph cluster.

[0075] Step 2: Use Kbuilder to quickly create Kubernetes' CRD: experimental resources and modeling resources, write core function experimental resource controllers and modeling resource controllers, the experimental resource controllers monitor the status changes of experimental resources and modeling resources and respond, and the modeling resource controllers monitor Kubernetes native Job resources and respond;

[0076] The experiment resource CRD has the following fields: experiment type, search space, optimizer configuration, total number of modeling, maximum experiment execution time, number of modelings executed in parallel, and maximum number of modeling failures;

[0077] The modeling resource CRD has the following fields: pause, resume, abandon, and whether to retain the Job resource after completion;

[0078] Step 3: Create a Manager service to receive modeling experiment requests;

[0079] Step 4: Verify the parameters of the experimental request and save it to MySQL;

[0080] Step 5: Create a Scheduling service to obtain the Mysql experiment request, and call Kubernetes to create experimental resources according to the experiment request;

[0081] Because experimental resources are created, the experimental resource controller will respond at this time, create modeling resources according to the experimental resource content, and monitor the state changes of the modeling resources to respond. For example, if the request requires a total of 4 modeling resources and 2 modeling resources are executed at the same time, the experimental resource controller will first create 2 modeling resources and then create 2 more after waiting for completion;

[0082] Because modeling resources are created, the modeling resource controller will respond by creating Job resources based on the modeling resource content and monitoring the status changes of the Job resources to respond. Kubernetes will create corresponding Pod resources based on the Job resources.

[0083] Step 6: Create a Webhook and inject it into the download container and the collection container, which are used to download the data required by the algorithm container and collect the results of the algorithm container and report them to the Manager;

[0084] See also Figure 4 , the Pod resource consists of a storage module, a data screening module, a download container module, an algorithm container module, and a collection container module. The storage module is used to store all the data transmitted by the Job controller, and then the data screening module screens out only the data required by the algorithm container, which is then downloaded by the download container module and used by the algorithm container module. The collection container module collects the results of the algorithm container and reports them to the Manager, allowing the algorithm container to focus on modeling without having to deal with whether the data comes from Ceph or Hive, and where to report the results. Webhook can also modify the input parameters of the container in the Pod to dynamically change the algorithm hyperparameters;

[0085] In the seventh step, the Manager service receives feedback from the experiment resource controller and the modeling resource controller, including the current state of the experiment, the current state of the modeling, and the data after the experiment ends, and saves the results to Mysql and Ceph.

[0086] Docker is a cloud-native technology-based modeling experimental system. It uses an algorithm image: Docker is used to generate an algorithm image, and then the algorithm image is submitted to the warehouse, with an accessible algorithm image address; data address: Ceph or Hive is used to save the data, with an accessible data address; algorithm hyperparameters: initialized algorithm hyperparameters; modeling control configuration: the number of modeling times N in the experiment, the number of modelings that can be executed simultaneously, the longest execution time of the experiment, the longest time for a single modeling, the maximum number of modeling failures, etc.; submit the algorithm image, data address and modeling control configuration to the modeling experiment system; the modeling experiment system provides an interface for querying the progress and results of the experiment, as well as pausing, resuming and stopping a single modeling in the experiment, to test and expand the functions of cloud-native technology. Only simple configuration is required to improve the efficiency of the experiment, complete the experiment faster, and have more control over the modeling in the experiment, including pausing, resuming and stopping, etc., and can also be set The longest execution time of an experiment and the maximum number of modeling failures that can occur are set. In short, the system of the present invention can provide a stable, efficient and controllable environment for modeling experiments, create a Webhook, inject a download container and a collection container, which are respectively used to download the data required by the algorithm container and collect the results of the algorithm container and report them to the Manager; the Pod resource is composed of a storage module, a data screening module, a download container module, an algorithm container module and a collection container module. The storage module is used to store all the data transmitted by the Job controller, and then the data screening module screens out only the data required by the algorithm container, which is then downloaded by the download container module and used by the algorithm container module. The collection container module collects the results of the algorithm container and reports them to the Manager, allowing the algorithm container to focus on modeling without having to process whether the data comes from Ceph or Hive, and where to report the results. The Webhook can also modify the input parameters of the container in the Pod to dynamically change the algorithm hyperparameters.

[0087] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device.

[0088] The above description is only the preferred embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A modeling experiment system based on cloud native technology, characterized in that: The following steps are involved: S1: Algorithm image: Use Docker to generate an algorithm image from the algorithm, and then submit the algorithm image to the warehouse, which has an accessible algorithm image address; S2: Data address: Use Ceph or Hive to save the data, and there is an accessible data address; S3: Algorithm hyperparameters: initialized algorithm hyperparameters; S4: Modeling control configuration: the number of modeling in the experiment N, the number of modeling that can be executed simultaneously, the maximum execution time of the experiment, the maximum time of a single modeling, and the maximum number of modeling failures; S5: Submit the algorithm image, data address and modeling control configuration to the modeling experiment system; S6: The modeling experiment system provides an interface to query the progress and results of the experiment, as well as to pause, resume, and stop a single model in the experiment; The specific implementation steps of S1-S6 are as follows: S101: Build a cluster; S102: Use Kbuilder to quickly create Kubernetes CRDs: experimental resources and modeling resources, write core function experimental resource controllers and modeling resource controllers, the experimental resource controllers monitor the status changes of experimental resources and modeling resources and respond, and the modeling resource controllers monitor Kubernetes native Job resources and respond; S103: Create a Manager service to receive modeling experiment requests; S104: Verify the parameters of the experimental request and save them to MySQL; S105: Create a Scheduling service to obtain the Mysql experiment request, and call Kubernetes to create experiment resources according to the experiment request; S106: Create a Webhook and inject a download container and a collection container into it. These containers are used to download the data required by the algorithm container and collect the results of the algorithm container and report them to the Manager. This allows the algorithm container to focus on modeling without having to deal with whether the data comes from Ceph or Hive, or where to report the results. The Webhook can also modify the input parameters of the container in the Pod to dynamically change the algorithm hyperparameters. S107: The Manager service receives feedback results from the experiment resource controller and the modeling resource controller.

2. A modeling experiment system based on cloud native technology as claimed in claim 1, characterized in that: The clusters built in S101 are Kubernetes cluster, MySQL cluster and Ceph cluster.

3. A modeling experiment system based on cloud native technology as described in claim 2, characterized in that: The experimental resource CRD in S102 has the following fields: experiment type, search space, optimizer configuration, total number of modeling, maximum experiment execution time, number of modelings executed in parallel, and maximum number of modeling failures.

4. A modeling experiment system based on cloud native technology as claimed in claim 3, characterized in that: The modeling resource CRD in S102 has the following fields: pause, resume, abandon, and whether to retain the Job resource after completion.

5. A modeling experiment system based on cloud native technology as described in claim 4, characterized in that: In S105, because experimental resources are created, the experimental resource controller will respond at this time, create modeling resources according to the experimental resource content, and monitor the status changes of the modeling resources to respond. The request requires a total of 4 modeling resources, and 2 modeling resources are executed at the same time. Then the experimental resource controller will first create 2 modeling resources, and then create 2 more after waiting for completion.

6. A modeling experiment system based on cloud native technology as claimed in claim 5, characterized in that: In S105, because modeling resources are created, the modeling resource controller will respond, create Job resources according to the modeling resource content, and respond by monitoring the status changes of the Job resources. Kubernetes creates corresponding Pod resources based on the Job resources.

7. A modeling experiment system based on cloud native technology as claimed in claim 6, characterized in that: The Pod resource consists of a storage module, a data screening module, a download container module, an algorithm container module, and a collection container module, which are interconnected.

8. A modeling experiment system based on cloud native technology as claimed in claim 7, characterized in that: The storage module is used to store all the data transmitted by the Job controller. The data screening module then screens out only the data required by the algorithm container, which is then downloaded by the download container module and used by the algorithm container module. The collection container module collects the results of the algorithm container and reports them to the Manager.

9. A modeling experiment system based on cloud native technology as claimed in claim 6, characterized in that: The feedback results in S107 include the current state of the experiment, the current state of the model and the data after the experiment is completed, and the results are saved in MySQL and Ceph.

Citation Information

Patent Citations

  • Multi-level deep learning log fault detection method for cloud native system

    CN111190804A

  • Machine learning model framework development method and system based on containerization technology

    CN112633501A