Artificial intelligence model construction method, system, device and storage medium

Abstract

The application relates to the technical field of artificial intelligence, and discloses an artificial intelligence model construction method, system, device and storage medium, the method comprising the following steps: constructing a container orchestration platform based on a distributed micro-service architecture, managing each virtual node in a cluster, and deploying an artificial intelligence algorithm library to the started virtual node; each component is spliced in a drag-and-drop form to generate instruction information for constructing a target artificial intelligence model, the instruction information comprising a directed acyclic graph formed between each component in the target artificial intelligence model; the instruction information is analyzed, the configuration resources required for constructing the target artificial intelligence model are determined according to the directed acyclic graph, the matched nodes are called based on the configuration resources to perform distributed training, and the target artificial intelligence model is generated; the application uses components to build an artificial intelligence model, has low threshold, is easy to construct, does not require code writing, and greatly improves the modeling efficiency; the component graphical modeling is clear and easy to understand.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a method, system, device and storage medium for constructing artificial intelligence models. Background Technology

[0002] With the development of science and technology, intelligentization is the current trend. Intelligentization is based on the automated processing of data, and the automated processing of data is inseparable from artificial intelligence models that process various types of data. Therefore, artificial intelligence models have been widely used in many fields, such as computer vision, image processing, natural language processing, information classification, search, recommendation, and big data, playing a significant role in promoting these fields. AI (Artificial Intelligence) models refer to mathematical models that use methods from mathematics, statistics, computer science, and machine learning to analyze, process, predict, and optimize data with certain regularities and predictability.

[0003] In related technologies, with the deepening application of artificial intelligence, more and more application scenarios require the use of artificial intelligence models to complete tasks such as data analysis, prediction, and classification. However, when building artificial intelligence models, on the one hand, it requires professionals with specialized knowledge to write code, which is not only costly but also time-consuming; on the other hand, it cannot meet the needs of various users and lacks versatility. This greatly increases the threshold for AI modeling and is not conducive to use by ordinary users. Summary of the Invention

[0004] To provide a basic understanding of some aspects of the disclosed embodiments, a brief summary is given below. This summary is not intended as a general commentary, nor is it intended to identify key / important components or describe the scope of protection of these embodiments, but rather as a prelude to the detailed description that follows.

[0005] In view of the shortcomings of the prior art described above, the present invention discloses an artificial intelligence model construction method, system, device and storage medium to overcome the problems of high threshold and difficulty in constructing artificial intelligence models.

[0006] In a first aspect, the present invention provides a method for constructing an artificial intelligence model, comprising:

[0007] A container orchestration platform is built based on a distributed microservice architecture to manage each virtual node in the cluster. An artificial intelligence algorithm library is deployed on each of the virtual nodes that are started. The artificial intelligence algorithm library includes data operators, training operators, algorithm operators and verification operators. Each type of operator encapsulates each algorithm in the form of a component. There is a one-to-one correspondence between the algorithm and the component.

[0008] By dragging and dropping the components together, instruction information for building a target artificial intelligence model is generated. The instruction information includes the service name of the target artificial intelligence model to be built and the directed acyclic graph formed between the components.

[0009] The system parses the instruction information, determines the configuration resources required to construct the target artificial intelligence model based on the directed acyclic graph, and calls the matching virtual nodes to perform distributed training based on the configuration resources to generate the target artificial intelligence model.

[0010] Optionally, the step of calling the matching virtual nodes based on the configured resources to perform distributed training and generate the target artificial intelligence model includes:

[0011] Obtain the current system resources of the cluster, including container resource configuration;

[0012] If the container resource configuration is greater than or equal to the configuration resources required by the target artificial intelligence model, then in response to the instruction information, each of the virtual nodes is invoked according to the configuration resources required by the target artificial intelligence model, so that each of the virtual nodes runs synchronously to distribute the training of each of the components and generate the target artificial intelligence model;

[0013] If the container resource configuration is less than the configuration resources required by the target artificial intelligence model, the container resource configuration is monitored in real time until the container resource configuration is greater than or equal to the configuration resources required by the target artificial intelligence model. In response to the instruction information, each of the virtual nodes is invoked according to the configuration resources required by the target artificial intelligence model, so that each of the virtual nodes runs synchronously to distribute the training of each of the components and generate the target artificial intelligence model.

[0014] Optionally, the step of parsing the instruction information in response to determine the configuration resources required to construct the target artificial intelligence model based on the directed acyclic graph, and calling the matching virtual nodes for distributed training based on the configuration resources, further includes:

[0015] If multiple instruction messages are received within a preset time, each instruction message is parsed to determine the service name of the artificial intelligence model corresponding to each instruction message;

[0016] The service names of the artificial intelligence models corresponding to each instruction information are sorted by priority to determine a priority list;

[0017] The configuration resources required by the target artificial intelligence model are determined in descending order of priority from the list, and the matching virtual nodes are called one by one in the order for distributed training.

[0018] If at least two AI models are found to have the same priority for their service names, arbitration is conducted based on the timestamps carried by the service names of the at least two AI models, and a first-come, first-served training method is used to construct the target AI model.

[0019] Optionally, the step of determining the configuration resources required by the target artificial intelligence model in descending order according to the priority list, and calling the matching virtual nodes one by one in the order for distributed training, includes:

[0020] The highest priority in the priority list is designated as the highest priority, and the lowest priority is designated as the lowest priority.

[0021] First, configure resources for the highest priority target AI model. After configuration, if there are still remaining resources in the current system, configure them one by one in order of priority from high to low, until the remaining resources in the current system are zero or the lowest priority target AI model is configured.

[0022] Each target artificial intelligence model is decomposed into multiple processing flows according to the corresponding directed acyclic graph, and each processing flow supports synchronous operation to achieve distributed training;

[0023] If it is determined that multiple processing flows from multiple target AI models simultaneously request resources from the current system, arbitration is performed based on the priority of the target AI model corresponding to each processing flow to determine the execution resource order of the processing flows.

[0024] Optionally, by dragging and dropping the components, instruction information for building the target artificial intelligence model is generated. This instruction information includes the service name of the target artificial intelligence model to be built and the directed acyclic graph formed between the components, including:

[0025] Each component in the artificial intelligence algorithm library is displayed in the tool component area in the form of a text representation, and the target component in the tool component area is selected by dragging and dropping.

[0026] Move the target component to the modeling canvas area, connect each component layer by layer to form a directed acyclic graph; determine the service name of the target artificial intelligence model based on the description information and control parameters of each component and the topology of the directed acyclic graph;

[0027] The service name of the target artificial intelligence model is combined with a directed acyclic graph to generate instruction information for constructing the target artificial intelligence model.

[0028] Optionally, the artificial intelligence model construction method further includes: constructing a distributed microservice architecture based on a Kubernetes cluster, wherein the Kubernetes cluster adopts a distributed file system combined cluster as a unified object storage solution; responding to the instruction information of the target artificial intelligence model, using the AVES service to parse the instruction information and determine the directed acyclic graph formed between the components in the instruction information; and calling application programming interface services in an orderly manner according to the directed acyclic graph formed between the components, distributively calling each program according to the order of data import, data processing, model training, model verification, and model saving to complete the construction of the target artificial intelligence model.

[0029] Optionally, the artificial intelligence model construction method further includes:

[0030] Select the data type operators that match the target artificial intelligence model and import them into the dataset to form the target dataset;

[0031] The target dataset is segmented into a training set and a validation set according to a preset ratio.

[0032] The training set and the validation set are preprocessed using the training class operator, and the preprocessing includes at least one of missing value imputation, duplicate row removal, random sampling, and data type conversion.

[0033] The preprocessed training set and the validation set are subjected to feature processing, which includes at least one of feature normalization, feature standardization, outlier handling, one-hot encoded data conversion, and wrapper feature selection.

[0034] In the algorithm class of operators, at least one operator that matches the target artificial intelligence model is selected, and the feature-processed training set is jointly trained with the at least one operator to construct the target artificial intelligence model;

[0035] In the verification operators, select the operator that matches the target artificial intelligence model for verification, determine the performance of the target artificial intelligence model, and continue until the performance of the target artificial intelligence model meets the preset index, and then output the target artificial intelligence model.

[0036] In a second aspect, the present invention provides an artificial intelligence model building system, comprising:

[0037] The operator configuration module is used to build a container orchestration platform based on a distributed microservice architecture, manage each virtual node in the cluster, and deploy an artificial intelligence algorithm library on each of the started virtual nodes. The artificial intelligence algorithm library includes data operators, training operators, algorithm operators and verification operators. Each type of operator encapsulates each algorithm in the form of a component, and there is a one-to-one correspondence between the algorithm and the component.

[0038] The instruction determination module is used to generate instruction information for building a target artificial intelligence model by assembling the components through drag and drop. The instruction information includes the service name of the target artificial intelligence model to be built and the directed acyclic graph formed between the components.

[0039] The model building module is used to parse the instruction information in response to the instruction information, determine the configuration resources required to build the target artificial intelligence model according to the directed acyclic graph, and call the matching virtual nodes to perform distributed training based on the configuration resources to generate the target artificial intelligence model.

[0040] In a third aspect, the present invention provides an electronic device, comprising: a processor and a memory; the memory for storing a computer program, and the processor for executing the computer program stored in the memory to cause the electronic device to perform the above-described method.

[0041] In a fourth aspect, the present invention provides a computer-readable medium having a computer program stored thereon, the computer program being used to cause a computer to perform the methods described above.

[0042] The beneficial effects of this invention are:

[0043] This invention constructs a container orchestration platform based on a distributed microservice architecture, managing virtual nodes within the cluster. An artificial intelligence algorithm library is deployed on each started virtual node. Each algorithm in the AI ​​algorithm library is encapsulated as a corresponding component. These components are assembled through drag-and-drop, generating instruction information for building a target AI model. The system parses this instruction information and determines the required configuration resources for building the target AI model based on a directed acyclic graph. Based on these configuration resources, the system calls the matching virtual nodes for distributed training to generate the target AI model. This approach achieves two advantages: firstly, it employs a graphical interface, using components to build AI models, which is low-threshold, easy to construct, and eliminates the need for coding, greatly improving the efficiency of AI model building; secondly, the graphical modeling of components not only makes the node processes clear and understandable but also enhances comprehensibility. Attached Figure Description

[0044] Figure 1 This is a schematic flowchart illustrating an exemplary embodiment of the present invention for a method of constructing an artificial intelligence model;

[0045] Figure 2 This is a business structure diagram illustrating an artificial intelligence model construction method according to an exemplary embodiment of the present invention;

[0046] Figure 3This is an example embodiment of the present invention illustrating the operation interface of an artificial intelligence model construction method;

[0047] Figure 4 This is a schematic diagram illustrating the structure of an artificial intelligence model building system according to an exemplary embodiment of the present invention.

[0048] Figure 5 This is a schematic diagram illustrating the implementation structure of an artificial intelligence model building system according to an exemplary embodiment of the present invention;

[0049] Figure 6 This is a schematic diagram illustrating the structure of a computer system suitable for implementing the electronic device of the present invention, as shown in an exemplary embodiment of the present invention. Detailed Implementation

[0050] The following specific examples illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that, unless otherwise specified, the following embodiments and sub-samples in the embodiments can be combined with each other.

[0051] It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of the present invention. Therefore, the drawings only show the components related to the present invention and are not drawn according to the actual number, shape and size of the components in the actual implementation. In the actual implementation, the form, quantity and proportion of each component can be arbitrarily changed, and the layout of the components may also be more complex.

[0052] In the following description, numerous details are explored to provide a more thorough explanation of embodiments of the invention. However, it will be apparent to those skilled in the art that embodiments of the invention may be practiced without these specific details. In other embodiments, well-known structures and devices are shown in block diagram form rather than in detail to avoid obscuring embodiments of the invention.

[0053] The terms "first," "second," etc., in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate for the embodiments of this disclosure described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. Unless otherwise stated, the term "a plurality of" means two or more. In this disclosure, the character " / " indicates that the preceding and following objects are in an "or" relationship. For example, A / B means: A or B. The term "and / or" describes an association between objects, indicating that three relationships can exist. For example, A and / or B means: A or B, or, A and B.

[0054] Please see Figure 1 This is a schematic flowchart illustrating an artificial intelligence model construction method according to an exemplary embodiment of the present invention. (Combined with...) Figure 1 As shown, in an exemplary embodiment, the artificial intelligence model construction method includes at least steps S101 to S103, which are described in detail below:

[0055] Step S101: Build a container orchestration platform based on a distributed microservice architecture to manage each virtual node in the cluster. Deploy an artificial intelligence algorithm library on each of the started virtual nodes. The artificial intelligence algorithm library includes data operators, training operators, algorithm operators and verification operators. Each type of operator encapsulates each algorithm in the form of a component, and there is a one-to-one correspondence between the algorithm and the component.

[0056] Specifically, distributed microservice architecture is a software development and deployment pattern that breaks down complex applications into small, independent, autonomous services. These services can be developed, deployed, and scaled independently, and can communicate and collaborate over a network. Microservice architecture can improve the reliability, scalability, and flexibility of applications, while reducing application maintenance costs and complexity.

[0057] In this system, the physical nodes of each server in the server cluster do not deploy algorithm libraries themselves. The server cluster is managed by Kubernetes. When the model is built and training begins, the system calls Kubernetes to start virtual nodes (Pods) in real time, so that the started virtual nodes contain artificial intelligence algorithm libraries.

[0058] See details Figure 2 The following is a business structure diagram of an artificial intelligence model construction method illustrated in an exemplary embodiment of the present invention, including:

[0059] By utilizing the various operator components (data operators, training operators, algorithm operators, and verification operators) of the system of this invention, AI models required for business scenarios can be easily constructed. Examples include industrial BMA four-layer sheet metal welding voltage parameter recommendation models, ME two-layer sheet metal welding pre-pressure time parameter recommendation models, ME two-layer sheet metal welding current parameter recommendation models, automotive door handle trim detection models, license plate size detection models, stamped part hole detection models, and keyless entry detection models for automobiles.

[0060] For example, data operators include, but are not limited to: machine learning data import, image classification data import, text classification data import, machine learning data segmentation, and image data segmentation;

[0061] Training operators include, but are not limited to: feature engineering control, regression training control, image classification training control, character recognition control, word vector training control, etc.

[0062] Algorithm operators include, but are not limited to: GBDT (Decision Tree) algorithm, Adboost (Iterative) algorithm, ResNet (Residual Neural Network) algorithm, Faster R-CNN (Object Detection) algorithm, word2vec (Word-to-Word Detection) algorithm, etc.

[0063] Validation operators include, but are not limited to: regression validation control, text detection validation control, object detection validation control, image classification validation control, word vector validation control, etc.

[0064] In this system, each algorithm in the various operators is encapsulated in a component. That is, each component represents an algorithm. It should be noted that the components are not limited to basic components or extended components.

[0065] Optionally, it should also be noted that the algorithm can also be encapsulated as a component in the following ways, as detailed below:

[0066] Metadata is defined for each algorithm, including: the format data of the input and output of the corresponding algorithm, and the format data of the parameters; based on the defined metadata, at least one of the algorithms is encapsulated to obtain the corresponding algorithm components.

[0067] Specifically, the algorithms can be based on the same computing framework or different computing frameworks. When defining metadata, the metadata for each algorithm can be defined separately; alternatively, some parameters that can be shared can be defined as metadata, which can reduce workload and facilitate metadata reuse.

[0068] After algorithm modularization, different algorithm components may need to transfer data. To meet this requirement, some important metadata in the algorithm, such as the format data of the input and output data and the format data of the parameters, needs to be defined separately. Generally, it is necessary to ensure that the format of the output of the algorithm component matches the format of the input of the algorithm component receiving the data. This matching can mean that the output format is the same as the input format or can be automatically converted.

[0069] For example, if an algorithm component needs to connect with two other algorithm components, which receive JSON data and binary data respectively, then when initially defining the algorithm component, the data input and output formats need to be defined, including JSON format and binary format.

[0070] When encapsulating algorithms, depending on the specific circumstances, one algorithm can be encapsulated as a single algorithm component, or multiple algorithms can be encapsulated as a single algorithm component. For example, if an algorithm is frequently used alone, or its combination with other algorithms is not fixed, it can be encapsulated as a separate algorithm component. Conversely, if an algorithm is often used in fixed combinations with one or more other algorithms, these algorithms can be encapsulated together to obtain an algorithm component containing multiple algorithms. This makes subsequent modeling more convenient and simplifies the model structure.

[0071] Step S102: By dragging and dropping, the components are assembled to generate instruction information for building the target artificial intelligence model. The instruction information includes the service name of the target artificial intelligence model and the directed acyclic graph formed between the components in building the target artificial intelligence model.

[0072] Please see Figure 3 An exemplary embodiment of the present invention illustrates an operation interface diagram of an artificial intelligence model construction method, which is described in detail below:

[0073] The components in the artificial intelligence algorithm library are displayed in the tool component area in the form of text representations, and the target component in the tool component area can be selected by dragging and dropping.

[0074] Move the target component to the modeling canvas area, connect the components layer by layer to form a directed acyclic graph; determine the service name of the target artificial intelligence model based on the description information of each component, the control parameters, and the topology of the directed acyclic graph.

[0075] The service name of the target AI model is combined with the directed acyclic graph to generate instruction information for constructing the target AI model.

[0076] Specifically, the AI ​​modeling operation page is divided into three main sections:

[0077] The Tools and Components area, located on the left side of the page, provides the basic components needed to build AI models, including various components for data, algorithms, training, and validation, totaling over 180 operator components (i.e., components). By using these tools and components, users no longer need to write tedious code; they can focus solely on their business logic to quickly build AI models.

[0078] Modeling Canvas Area: Located in the middle of the page, you can drag and drop operator components from the left-hand tool component area onto the canvas, and connect multiple components layer by layer to build a directed acyclic graph to achieve AI modeling.

[0079] Component configuration description area: Located on the right side of the page, it contains descriptions of the components selected by the user and a form showing a series of adjustable parameters for the components, so that users can understand the components and adjust the parameters of AI modeling.

[0080] The above method not only determines the service name of the target AI model, but also combines the service name of the target AI model with a directed acyclic graph to generate instruction information containing the construction of the target AI model, so that the target AI model can be constructed quickly and accurately by responding directly to the instruction information.

[0081] Step S103: In response to the instruction information, the configuration resources required to build the target artificial intelligence model are determined according to the directed acyclic graph. Based on the configuration resources, the matching nodes are called to perform distributed training and generate the target artificial intelligence model.

[0082] It should be noted that container orchestration platforms include, but are not limited to, Docker Swarm (cluster management tool), Nomad (cluster management tool), Elastic Container Service (cluster management tool), Onteon (cluster management tool), and Kubernetes (cluster management tool). In this embodiment, Kubernetes is used as an example. Kubernetes has become an industry benchmark and is free and open source.

[0083] Of course, this invention also develops a distributed container orchestration platform for building AI modeling systems, but distributed development is difficult and costly. Regarding the upper-layer web system, you can use programming languages ​​such as C++ and Python to develop web (web page) systems, but Python has lower performance than Java, and C++ has lower development efficiency. For the upper-layer AI modeling framework, PyTorch, TensorFlow, and Scikit-Learn are the primary choices. MxNet, Caffe, and PaddlePaddle can also be used as alternatives, but they are not mainstream and are not conducive to users developing custom AI components or troubleshooting.

[0084] In this embodiment, the computation of a single operator in the target artificial intelligence model is distributed across multiple hardware devices for concurrent computation, thereby improving the computation speed of a single operator. Specifically, the computation of a single operator is distributed across several identically configured hardware devices using a model-parallel approach for model storage and computation, ensuring consistent computational pace. Distributed training not only offers better performance and higher upper limits on runtime resources but also fully utilizes cluster resources and reduces overhead.

[0085] The above method constructs a container orchestration platform based on a distributed microservice architecture to manage virtual nodes within the cluster. An artificial intelligence algorithm library is deployed on each started virtual node. Each algorithm in the library is encapsulated as a corresponding component. These components are assembled through drag-and-drop, generating instruction information for building the target artificial intelligence model. This instruction information is parsed, and the required configuration resources for building the target artificial intelligence model are determined based on a directed acyclic graph. Based on these configuration resources, the matching virtual nodes are invoked for distributed training to generate the target artificial intelligence model. This approach achieves two advantages: firstly, it employs a graphical interface and utilizes components to build AI models, which is low-threshold, easy to construct, and eliminates the need for coding, greatly improving the modeling efficiency of artificial intelligence models; secondly, the graphical modeling of components not only makes the node processes clear and understandable but also enhances comprehensibility.

[0086] Optionally, distributed training is performed based on nodes matching the configured resource calls to generate the target artificial intelligence model, including:

[0087] Get the current system resources of the cluster, including container resource configuration;

[0088] If the container resource configuration is greater than or equal to the configuration resources required by the target artificial intelligence model, then in response to the instruction information, each node is invoked to respond according to the configuration resources required by the target artificial intelligence model, so that each node distributes the training of each component and generates the target artificial intelligence model.

[0089] If the container resource configuration is less than the configuration resources required by the target artificial intelligence model, the container resource configuration is monitored in real time until the container resource configuration is greater than or equal to the configuration resources required by the target artificial intelligence model. In response to the instruction information, each node is called to respond according to the configuration resources required by the target artificial intelligence model, so that each node can distribute the training of each component and generate the target artificial intelligence model.

[0090] In this embodiment, the current system resources of the cluster are monitored in real time, and the current system resources of the cluster are compared with the configuration resources required by the target artificial intelligence model. In this way, it can be quickly determined whether the target artificial intelligence model can be built. This avoids the system lag and slowdown caused by calling resources to build the target artificial intelligence model when resources are insufficient, which would affect the efficiency of model building. By adopting the above method, when container resources are insufficient, the model building task can only be suspended, without affecting the smooth operation of the entire system. This greatly improves the efficiency of building the target artificial intelligence model and enhances the user experience.

[0091] Optionally, the step of parsing the instruction information in response to determine the configuration resources required to construct the target artificial intelligence model based on the directed acyclic graph, and calling the matching virtual nodes for distributed training based on the configuration resources, further includes:

[0092] If multiple instruction messages are received within a preset time, each instruction message is parsed to determine the service name of the artificial intelligence model corresponding to each instruction message;

[0093] The service names of the artificial intelligence models corresponding to each instruction information are sorted by priority to determine a priority list;

[0094] The configuration resources required by the target artificial intelligence model are determined in descending order of priority from the list, and the matching virtual nodes are called one by one in the order for distributed training.

[0095] If at least two AI models are found to have the same priority for their service names, arbitration is conducted based on the timestamps carried by the service names of the at least two AI models, and a first-come, first-served training method is used to construct the target AI model.

[0096] In the above manner, when multiple instruction messages are received at the same time, arbitration is carried out according to priority to determine which target artificial intelligence model to allocate resources first. If there are at least two artificial intelligence models with the same priority, arbitration is carried out based on the timestamps, and the target artificial intelligence model is constructed by first-come-first-served training.

[0097] Optionally, the step of determining the configuration resources required by the target artificial intelligence model in descending order according to the priority list, and calling the matching virtual nodes one by one in the order for distributed training, includes:

[0098] The highest priority in the priority list is designated as the highest priority, and the lowest priority is designated as the lowest priority.

[0099] First, configure resources for the highest priority target AI model. After configuration, if there are still remaining resources in the current system, configure them one by one in order of priority from high to low, until the remaining resources in the current system are zero or the lowest priority target AI model is configured.

[0100] Each target artificial intelligence model is decomposed into multiple processing flows according to the corresponding directed acyclic graph, and each processing flow supports synchronous operation to achieve distributed training;

[0101] If it is determined that multiple processing flows from multiple target AI models simultaneously request resources from the current system, arbitration is performed based on the priority of the target AI model corresponding to each processing flow to determine the execution resource order of the processing flows.

[0102] In this embodiment, instead of applying for all resources for the entire target AI model, cluster resources are managed in a fine-grained manner for each of the processing flows, which avoids resource waste and improves the concurrency capability of the cluster training model.

[0103] Optionally, the artificial intelligence model construction method also includes: building a distributed microservice architecture based on a Kubernetes cluster, wherein the Kubernetes cluster adopts a distributed file system combined cluster as a unified object storage solution; responding to the instruction information of the target artificial intelligence model, using the AVES service to parse the instruction information and determine the directed acyclic graph formed between the components in the instruction information; and calling the application interface services in an orderly manner according to the directed acyclic graph formed between the components, distributing the artificial intelligence model to call each program in the order of data import, data processing, model training, model verification, and model saving to complete the construction of the target artificial intelligence model.

[0104] Specifically, Kubernetes is an open-source container orchestration engine used to automate the deployment, scaling, and management of containerized applications. It combines multiple physical servers into a large virtualization system. On this system, Pods (which can be understood as virtual servers that can request resources such as CPU, memory, and disk) can be virtualized. Specific programs, such as Java programs or MySQL databases, can run within these Pod containers.

[0105] For example, Ceph refers to a distributed system that provides block storage, file system, and object storage services.

[0106] For example, Spring Cloud refers to a web framework in Java programming, Feign refers to a remote HTTP request and invocation framework in Java programming, and Zuul refers to a gateway routing framework in Java programming.

[0107] For example, the deep learning and machine learning frameworks used in this system include PyTorch, TensorFlow, and Scikit-Learn.

[0108] In this embodiment, the present invention provides a low-code, component-based AI modeling solution. This solution, by employing technologies such as Kubernetes, virtualization, microservices, CUDA (Graphics Processing Development Environment), machine learning, and deep learning, significantly improves overall modeling concurrency and performance, reduces unnecessary resource consumption, and provides users with a wealth of reusable components. Users can select appropriate components according to their actual needs to quickly build AI algorithm models. The distributed, component-based, and low-code AI modeling capabilities provided by this invention solve the problems of high AI modeling barriers, insufficient utilization of AI training resources, and single-node bottlenecks in AI training.

[0109] Alternatively, artificial intelligence model building methods also include:

[0110] Select the data operators that match the target artificial intelligence model from the data type operators and import them into the dataset to form the target dataset;

[0111] The target dataset is split into a training set and a validation set according to a preset ratio.

[0112] The training set and the validation set are preprocessed using the training class operator, and the preprocessing includes at least one of missing value imputation, duplicate row removal, random sampling, and data type conversion.

[0113] The preprocessed training set and the validation set are subjected to feature processing, which includes at least one of feature normalization, feature standardization, outlier handling, one-hot encoded data conversion, and wrapper feature selection.

[0114] In the algorithm class operators, select at least one operator that matches the target artificial intelligence model, and jointly train the feature-processed training set with at least one operator to construct the target artificial intelligence model;

[0115] In the verification operators, select operators that match the target AI model for verification, determine the performance of the target AI model, and continue until the performance of the target AI model meets the preset indicators, and then output the target AI model.

[0116] Specifically, in Figure 3 In the middle is an AI modeling example of a "dimensional tolerance correlation analysis model," and its operation steps are as follows:

[0117] First, the "Dimensional Coordinate Measurement Dataset" is imported by using the "Machine Learning Data Import" operator.

[0118] Next, the "machine learning data segmentation" operator is used to divide the dimensional coordinate measurement data into a training set and a validation set.

[0119] Next, data preprocessing is performed by adding operators such as "fill missing values", "remove duplicate rows" and "data type conversion". The "data preprocessing control" operator is used to connect the data and processing operators together to realize the data preprocessing function.

[0120] Next, feature engineering is performed, adding operators such as "feature normalization", "outlier handling", "BoxCox transformation" and "wrap-style feature selection", and using the "feature engineering control" operator to connect the data and processing operators together to realize the feature engineering function.

[0121] Then, select the "LASSO regression" operator, the basic algorithm required for this "dimensional tolerance correlation analysis" model, and use the "regression training control" operator to connect the data to be trained and the algorithm together to realize the AI ​​model training function.

[0122] Finally, drag in the "Regression Validation Control" operator to connect the data to be validated and the trained model together, thus realizing the evaluation function of the AI ​​model.

[0123] Through the above methods, the AI ​​modeling solution of this invention is based on Kubernetes, which is a benchmark for distributed virtualization platforms in the industry. Relatively speaking, it possesses better robustness, stability, security, versatility, and concurrency performance. For example, the underlying system has built-in user security authentication, not just at the upper-layer web application, providing more granular security protection. For example, if any virtual node in the cluster fails, other virtual nodes will be automatically started to replace it and resume the training task, demonstrating excellent robustness. Furthermore, the underlying Kubernetes system has been widely used in many companies (Google, Alibaba, Tencent, Huawei, Baidu, etc.), ensuring the system's broad applicability, greater versatility, and ease of maintenance.

[0124] Please see Figure 4 This is a schematic diagram illustrating the structure of an artificial intelligence model building system, as shown in an exemplary embodiment of the present invention. (Combined with...) Figure 4 As shown, the exemplary artificial intelligence model building system 400 includes: an operator configuration module 401, an instruction determination module 402, and a model building module 405, wherein:

[0125] The operator configuration module 401 is used to build a container orchestration platform based on a distributed microservice architecture, manage each virtual node in the cluster, and deploy an artificial intelligence algorithm library on each virtual node that is started. The artificial intelligence algorithm library includes data operators, training operators, algorithm operators and verification operators. Each type of operator encapsulates each algorithm in the form of a component, and there is a one-to-one correspondence between the algorithm and the component.

[0126] The instruction determination module 402 is used to generate instruction information for building a target artificial intelligence model by assembling various components through drag and drop. The instruction information includes the service name of the target artificial intelligence model and the directed acyclic graph formed between the various components in building the target artificial intelligence model.

[0127] The model building module 403 is used to parse the instruction information in response, determine the configuration resources required to build the target artificial intelligence model based on the directed acyclic graph, call the matching nodes based on the configuration resources to perform distributed training, and generate the target artificial intelligence model.

[0128] It should be noted that the artificial intelligence model building system provided in the above embodiments and the artificial intelligence model building method provided in the above embodiments belong to the same concept. The specific way of performing each step has been described in detail in the system embodiments, and will not be repeated here.

[0129] The artificial intelligence model building system provided in this embodiment of the present disclosure builds a container orchestration platform based on a distributed microservice architecture to manage virtual nodes within the cluster. Artificial intelligence algorithm libraries are deployed on each of the started virtual nodes. Various algorithms in the artificial intelligence algorithm library are encapsulated as corresponding components. These components are assembled through drag-and-drop to generate instruction information for building the target artificial intelligence model. The system parses the instruction information and determines the configuration resources required to build the target artificial intelligence model based on a directed acyclic graph. Based on the configuration resources, matching nodes are called for distributed training to generate the target artificial intelligence model. Through this method, on the one hand, the use of graphical operations and components to build AI models is low-threshold, easy to build, and eliminates the need for coding, greatly improving the modeling efficiency of artificial intelligence models; on the other hand, the graphical modeling of components not only makes the node processes clear and understandable but also enhances comprehensibility.

[0130] Please see Figure 5 The diagram illustrates an implementation structure of an artificial intelligence model building system, as shown in an exemplary embodiment of the present invention, comprising:

[0131] A Kubernetes cluster is built based on a distributed microservice architecture. The Kubernetes cluster uses a distributed file system combined with the cluster as a unified object storage solution. In response to the instruction information of the target artificial intelligence model, the AVES service is used to parse the instruction information and determine the directed acyclic graph formed between the components in the instruction information. According to the directed acyclic graph formed between the components, the application interface services are called in an orderly manner. The artificial intelligence model is distributed and calls the programs in the order of data import, data processing, model training, model verification and model saving to complete the construction of the target artificial intelligence model.

[0132] Specifically, Kubernetes virtualized clusters are used as the service foundation, Ceph composite clusters are used as a unified object storage solution, and system service Pods are built on this basis, including AVES service Pods, model-publish service Pods, business service Pods, etc.

[0133] Based on the user's design of the AI ​​model, the system dynamically starts the data import Pod (i.e., imports data from a remote source and then stores it in distributed object storage), splits the data in the distributed object storage to form training and validation sets, stores them in distributed object storage again, reads the training and validation sets for preprocessing and preprocessing control, the AI ​​algorithm training Pod (reads the preprocessed training set for training to determine the AI ​​model), and the validation control Pod (reads the preprocessed validation set to verify the AI ​​model), etc.

[0134] Upon receiving the user's AI model building task (i.e., instruction information), the AVES service sends it to the AVES service. The AVES service parses the task information and, based on the user's constructed AI model DAG (Directed Acyclic Graph), sequentially calls the Kubernetes API Server (Application Programming Interface Service) to launch Pods (programs) in a distributed manner, and executes the AI ​​model building process step by step (including data import, data processing, model training, model evaluation, and model saving) to ultimately complete the modeling task.

[0135] When a user needs to build an AI model before using it, the corresponding pre-trained AI model will be retrieved from Ceph object storage. The distributed storage capability will be used to store a large number of models, data, operator components, etc.

[0136] Please see Figure 6 A schematic diagram of a computer system suitable for implementing embodiments of the present invention is shown. It should be noted that... Figure 6 The computer system 600 of the electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of the present invention.

[0137] like Figure 6 As shown, the computer system 600 includes a Central Processing Unit (CPU) 601, which can perform various appropriate actions and processes, such as executing the methods described in the above embodiments, based on programs stored in Read-Only Memory (ROM) 602 or programs loaded from Storage Unit 608 into Random Access Memory (RAM) 603. The RAM 603 also stores various programs and data required for system operation. The CPU 601, ROM 602, and RAM 603 are interconnected via a bus 304. An Input / Output (I / O) interface 605 is also connected to the bus 604.

[0138] The following components are connected to I / O interface 605: an input section 606 including a keyboard, mouse, etc.; an output section 607 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 608 including a hard disk, etc.; and a communication section 609 including a network interface card such as a LAN (Local Area Network) card, modem, etc. The communication section 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to I / O interface 605 as needed. A removable medium 611, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on drive 610 as needed so that computer programs read from it can be installed into storage section 608 as needed.

[0139] In particular, according to embodiments of the present invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of the present invention include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing computer programs for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 609, and / or installed from removable medium 611. When the computer program is executed by central processing unit (CPU) 601, it performs various functions defined in the system of the present invention.

[0140] The present invention also provides a computer-readable and writable storage medium storing a computer program, which, when executed, implements at least one embodiment described above for the method of constructing an artificial intelligence model, such as... Figure 1 The described embodiments.

[0141] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods of the various embodiments of the present invention.

[0142] In the embodiments provided by this invention, a computer-readable and writable storage medium may include a read-only memory, random access memory, EEPROM, CD-ROM or other optical disc storage devices, disk storage devices or other magnetic storage devices, flash memory, USB flash drive, portable hard drive, or any other medium capable of storing desired program code having an instruction or data structure form and accessible by a computer. Additionally, any connection may be appropriately referred to as a computer-readable medium. For example, if the instructions are transmitted from a website, server, or other remote source using coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwave, then coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave are included in the definition of medium. However, it should be understood that computer-readable and writable storage media and data storage media do not include connections, carrier waves, signals, or other transient media, but are intended for non-transient, tangible storage media. The disks and optical discs used in the application include compact discs (CDs), laser discs, optical discs, digital multifunction discs (DVDs), floppy disks, and Blu-ray discs, where disks typically copy data magnetically, while optical discs use lasers to copy data optically.

[0143] In one or more exemplary aspects, the functionality described by the computer program of the method of the present invention can be implemented in hardware, software, firmware, or any combination thereof. When implemented in software, these functions can be stored or transmitted as one or more instructions or code onto a computer-readable medium. The steps of the method or algorithm disclosed in this invention can be embodied in a processor-executable software module, wherein the processor-executable software module can reside on a tangible, non-transitory computer-readable and writable storage medium. The tangible, non-transitory computer-readable and writable storage medium can be any available medium accessible to a computer.

[0144] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the invention. Accordingly, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented by a dedicated hardware-based system that performs the specified function or operation, or by a combination of dedicated hardware and computer instructions.

[0145] The above embodiments are merely illustrative of the principles and effects of the present invention and are not intended to limit the invention. Any person skilled in the art can modify or alter the above embodiments without departing from the spirit and scope of the present invention. Therefore, all equivalent modifications or alterations made by those skilled in the art without departing from the spirit and technical concept disclosed in the present invention should still be covered by the claims of the present invention.

Claims

1. A method for constructing an artificial intelligence model, characterized in that, include: A container orchestration platform is built based on a distributed microservice architecture to manage each virtual node in the cluster. An artificial intelligence algorithm library is deployed on each of the virtual nodes that are started. The artificial intelligence algorithm library includes data operators, training operators, algorithm operators and verification operators. Each type of operator encapsulates each algorithm in the form of a component. There is a one-to-one correspondence between the algorithm and the component. By dragging and dropping the components together, instruction information for building a target artificial intelligence model is generated. The instruction information includes the service name of the target artificial intelligence model to be built and the directed acyclic graph formed between the components. In response to the instruction information, the system parses the data, determines the configuration resources required to construct the target artificial intelligence model based on the directed acyclic graph, and performs distributed training by calling the matching virtual nodes based on the configuration resources to generate the target artificial intelligence model, including: If multiple instruction messages are received within a preset time, each instruction message is parsed to determine the service name of the artificial intelligence model corresponding to each instruction message; The highest priority model in the priority list is designated as the highest priority, and the lowest priority model is designated as the lowest priority. Resources are first allocated to the highest priority target AI model. After allocation, if the system still has remaining resources, resources are allocated sequentially according to the priority list from highest to lowest, until the system has zero remaining resources or the lowest priority target AI model is fully allocated. Each target AI model is decomposed into multiple processing flows according to the corresponding directed acyclic graph. Each processing flow supports synchronous operation to achieve distributed training. If multiple processing flows from multiple target AI models simultaneously request resources from the current system, arbitration is performed based on the priority of the target AI model corresponding to each processing flow to determine the resource allocation order of the processing flows. If at least two AI models are found to have the same priority for their service names, arbitration is conducted based on the timestamps carried by the service names of the at least two AI models, and a first-come, first-served training method is used to construct the target AI model.

2. The artificial intelligence model construction method according to claim 1, characterized in that, The step of performing distributed training based on the matching virtual nodes according to the configured resources to generate the target artificial intelligence model includes: Obtain the current system resources of the cluster, including container resource configuration; If the container resource configuration is greater than or equal to the configuration resources required by the target artificial intelligence model, then in response to the instruction information, each of the virtual nodes is invoked according to the configuration resources required by the target artificial intelligence model, so that each of the virtual nodes runs synchronously to distribute the training of each of the components and generate the target artificial intelligence model; If the container resource configuration is less than the configuration resources required by the target artificial intelligence model, the container resource configuration is monitored in real time until the container resource configuration is greater than or equal to the configuration resources required by the target artificial intelligence model. In response to the instruction information, each of the virtual nodes is invoked according to the configuration resources required by the target artificial intelligence model, so that each of the virtual nodes runs synchronously to distribute the training of each of the components and generate the target artificial intelligence model.

3. The artificial intelligence model construction method according to claim 1, characterized in that, By dragging and dropping the components, instruction information for building a target artificial intelligence model is generated. This instruction information includes the service name of the target artificial intelligence model to be built and the directed acyclic graph formed between the components, including: Each component in the artificial intelligence algorithm library is displayed in the tool component area in the form of a text representation, and the target component in the tool component area is selected by dragging and dropping. Move the target component to the modeling canvas area, connect each component layer by layer to form a directed acyclic graph; determine the service name of the target artificial intelligence model based on the description information and control parameters of each component and the topology of the directed acyclic graph; The service name of the target artificial intelligence model is combined with a directed acyclic graph to generate instruction information for constructing the target artificial intelligence model.

4. The artificial intelligence model construction method according to any one of claims 1 to 3, characterized in that, A distributed microservice architecture is built on a Kubernetes cluster, which uses a distributed file system cluster as a unified object storage solution. In response to the instruction information of the target artificial intelligence model, the instruction information is parsed using the AVES service to determine the directed acyclic graph formed between the components in the instruction information; the application programming interface services are called in an orderly manner according to the directed acyclic graph formed between the components, and the artificial intelligence model is distributed to call each program in the order of data import, data processing, model training, model verification and model saving to complete the construction of the target artificial intelligence model.

5. The artificial intelligence model construction method according to any one of claims 1 to 3, characterized in that, Also includes: Select the data type operators that match the target artificial intelligence model and import them into the dataset to form the target dataset; The target dataset is segmented into a training set and a validation set according to a preset ratio. The training set and the validation set are preprocessed using the training class operator, and the preprocessing includes at least one of missing value imputation, duplicate row removal, random sampling, and data type conversion. The preprocessed training set and the validation set are subjected to feature processing, which includes at least one of feature normalization, feature standardization, outlier handling, one-hot encoding, and wrapper feature selection. In the algorithm class of operators, at least one operator that matches the target artificial intelligence model is selected, and the feature-processed training set is jointly trained with the at least one operator to construct the target artificial intelligence model; In the verification operators, select the operator that matches the target artificial intelligence model for verification, determine the performance of the target artificial intelligence model, and continue until the performance of the target artificial intelligence model meets the preset index, and then output the target artificial intelligence model.

6. An artificial intelligence model building system, characterized in that, include: The operator configuration module is used to build a container orchestration platform based on a distributed microservice architecture, manage each virtual node in the cluster, and deploy an artificial intelligence algorithm library on each of the started virtual nodes. The artificial intelligence algorithm library includes data operators, training operators, algorithm operators and verification operators. Each type of operator encapsulates each algorithm in the form of a component, and there is a one-to-one correspondence between the algorithm and the component. The instruction determination module is used to generate instruction information for building a target artificial intelligence model by assembling the components through drag and drop. The instruction information includes the service name of the target artificial intelligence model to be built and the directed acyclic graph formed between the components. The model building module is used to parse the instruction information in response, determine the configuration resources required to build the target artificial intelligence model based on the directed acyclic graph, and call the matching virtual nodes to perform distributed training based on the configuration resources to generate the target artificial intelligence model, including: If multiple instruction messages are received within a preset time, each instruction message is parsed to determine the service name of the artificial intelligence model corresponding to each instruction message; The highest priority model in the priority list is designated as the highest priority, and the lowest priority model is designated as the lowest priority. Resources are first allocated to the highest priority target AI model. After allocation, if the system still has remaining resources, resources are allocated sequentially according to the priority list from highest to lowest, until the system has zero remaining resources or the lowest priority target AI model is fully allocated. Each target AI model is decomposed into multiple processing flows according to the corresponding directed acyclic graph. Each processing flow supports synchronous operation to achieve distributed training. If multiple processing flows from multiple target AI models simultaneously request resources from the current system, arbitration is performed based on the priority of the target AI model corresponding to each processing flow to determine the resource allocation order of the processing flows. If at least two AI models are found to have the same priority for their service names, arbitration is conducted based on the timestamps carried by the service names of the at least two AI models, and a first-come, first-served training method is used to construct the target AI model.

7. An electronic device, characterized in that, include: Processor and memory; The memory is used to store a computer program, and the processor is used to execute the computer program stored in the memory to cause the electronic device to perform the method as described in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, It stores a computer program thereon, the computer program being used to cause the computer to perform the method as described in any one of claims 1 to 5.

Images