An autonomously controllable heterogeneous intelligent computing service platform and an intelligent scene matching method

Abstract

The application provides an autonomously controllable heterogeneous intelligent computing service platform and an intelligent scene matching method, which can integrate various complex fusion heterogeneous resources in the intelligent computing service platform and effectively manage the resources to meet the diverse intelligent computing acceleration needs of users; the intelligent scene matching method supports an intelligent model deployment and arrangement process based on scene characteristics, and can realize intelligent matching based on concurrency, timeliness, accuracy and business type scene characteristics; the method can also automatically deploy on the matched computing resources, and finally realize automatic deployment of intelligent applications in a production environment; the autonomously controllable vertical field intelligent computing service platform and the intelligent scene matching method can be well applied to scenes related to the field of artificial intelligence.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence technology and relates to an independent and controllable intelligent computing platform that uses domestically produced heterogeneous computing resources and can simultaneously perform machine learning model training, inference, and instantiation deployment, as well as a method for intelligent scene matching based on typical application scenarios. Background Technology

[0002] Currently, artificial intelligence technologies, represented by machine learning, are flourishing in various practical intelligent application scenarios. Various intelligent devices, such as intelligent access control and security cameras, are being used in a wide range of real-world scenarios using machine learning technology. The implementation of machine learning technology mainly involves using general-purpose computing resources such as CPU and GPU to integrate and process large amounts of data generated in real-world situations. By analyzing needs and data characteristics, a learning model that meets the target requirements is established and continuously trained to improve the accuracy of the learning model, enabling intelligent devices to more accurately reflect a higher level of intelligence.

[0003] With the continuous development of computer hardware and software technologies, among the main implementation methods mentioned above, from the perspective of computing power development trends, diverse and complex integrated computing resource architectures such as CPU+ASIC and CPU+FPGA have emerged, greatly improving the computing power for machine learning model operations. From the perspective of machine learning model development, models applying new technical systems have also appeared in multiple business areas, enhancing the ability of machine learning models to cope with complex intelligent scenarios. However, increasingly complex and diverse integrated computing resource architectures and different technical system models bring about problems such as resource management, resource scheduling, and model selection. Therefore, it is urgent to provide an intelligent computing service platform that can effectively manage diverse, complex, integrated, and heterogeneous resources and provide scenario-based intelligent matching services. Summary of the Invention

[0004] In view of this, the purpose of this invention is to provide an autonomous and controllable heterogeneous intelligent computing service platform and an intelligent scene matching method, which can integrate and effectively manage a variety of complex and heterogeneous resources into this intelligent computing service platform, meet the diverse intelligent computing acceleration needs of users, and ultimately realize the automated deployment of intelligent applications in the production environment.

[0005] An intelligent computing service platform with independent and controllable software and hardware includes a heterogeneous resource scheduling engine, an intelligent model training orchestration module, a unified heterogeneous resource management module, a comprehensive intelligent interactive front-end module, and an automatic application model deployment module.

[0006] The heterogeneous resource scheduling engine provides a unified resource scheduling framework for intelligent computing service platforms, enabling resource scheduling of various heterogeneous computing resources at the infrastructure level to meet the deployment requirements of intelligent applications.

[0007] The intelligent orchestration module for model training is used to customize the training process of machine learning models based on scene templates. The training deployment process can be intelligently matched according to the scene template selected by the user, and multiple training deployment schemes are provided for the user to choose from. Users can customize the data preprocessing, model training, and environment configuration process on the provided training deployment schemes.

[0008] The heterogeneous resource unified management module is used to manage various types of heterogeneous resources in a unified manner.

[0009] The integrated intelligent interactive front-end module is used for interaction between the intelligent computing service platform and users, supporting model development, training, and intelligent application inference;

[0010] The application model automatic deployment module is used to package and deploy pre-trained models developed on the platform to the production environment, directly providing intelligent services to users.

[0011] Preferably, the heterogeneous resource unified management module is deployed on the slave node; the heterogeneous resource scheduling engine, the model training intelligent orchestration module, the comprehensive intelligent interactive front-end module, and the application model automatic deployment module are deployed on the master node.

[0012] Preferably, the heterogeneous computing resources include GPUs, FPGAs, and intelligent ASIC chips; specifically, they include domestic CPUs Phytium and Shenwei, intelligent accelerator cards Cambricon, Bitmain, Huawei Ascend, Baidu Kunlun, and heterogeneous computing resources composed of combinations of CPUs, GPUs, FPGAs, and intelligent ASICs.

[0013] Preferably, the intelligent computing framework includes Tensorflow, PyTorch, ONNX, PaddlePaddle, and Jittor frameworks.

[0014] Preferably, the model training intelligent orchestration module allows users to select scenarios including image classification, object detection, NLP, speech recognition, speech synthesis, object recommendation, image generation, and text generation; trainable and developable models include GhostNet, DPN, SE_ResNeXt, Xception, EfficientNet, MaskR-CNN, ShapeMask, SpineNet, yoloV3_darknet5, DeepLab_v3_s16, LSTM, ERNIE, BERT, GCN, GAT, BLSTM, WaveNet, WaveGlow, Tacotron, RNN-T, NCF, GNN, DLRM, DeepFM, NFM, MiniGo, DCGAN, PGAN, DDPG, and DQN; and available training deployment schemes include high-performance schemes prioritizing performance and high-reliability schemes prioritizing reliability.

[0015] An intelligent scene matching method includes:

[0016] Step 1: Add a new heterogeneous resource node;

[0017] Step 2: Use the platform's built-in models and datasets to test the models. Execute the script to calculate the time and accuracy of model inference. Add the unit inference time and accuracy on this resource to the model's attribute table. Repeat this process for all built-in models on the platform to obtain the unit inference time and accuracy of all built-in models on this heterogeneous resource. Establish a mapping relationship between the model and all resource nodes based on the unit inference time and accuracy parameters.

[0018] Step 3: The user uploads a dataset of size D, inputs the timeliness target value T and the accuracy target value M, and selects the business type. Then, the range of model groups for this type is determined based on the business type label.

[0019] Step 4: First, remove all model-resource node mappings within the model group whose estimated inference time on all available heterogeneous resource nodes is greater than the timeliness target T or whose accuracy is less than the accuracy target M. Here, the estimated inference time = unit inference time * D. This yields all model-resource node mappings that meet the requirements, f: {m>s}, where m is the set of models that meet the requirements and s is the set of resource nodes that meet the requirements.

[0020] Step 5: Next, based on the node resource usage information reported from the nodes to the master node, nodes in s whose resource usage does not meet the actual deployment requirements are removed, resulting in a model and resource node mapping f':{m'>s'} that can be actually deployed, where f', m', and s' are subsets of f, m, and s, respectively.

[0021] Step Six: Finally, in the obtained model-resource node mapping f':{m'>s'}, sort them according to the accuracy parameter of each mapping relationship, and take the model-resource node mapping F with the highest accuracy as the algorithm output result;

[0022] Step 7: Based on the unit inference time t of the model-resource node mapping F, calculate the concurrency P = (D*t) / T. When P≤1, the number of instances is 1, and when P>1, the number of instances is [P]+1. Thus, the matching result is determined.

[0023] Preferably, in step one, a monitoring program and an agent program are deployed on the heterogeneous resource node, and the monitoring program is executed periodically to call system commands and board drivers to obtain the resource type and usage status of the heterogeneous resource node.

[0024] Preferably, in step four, if none of the models in the model group can meet the timeliness target T, then the concurrency P = expected inference time / T is calculated. At this time, all models and resource node mappings that are greater than T / P or less than M should be removed.

[0025] Preferably, in step four, the estimated inference time is the median of the estimated inference times of all models in the model group.

[0026] The present invention has the following beneficial effects:

[0027] The present invention provides an autonomous and controllable heterogeneous intelligent computing service platform that can integrate and effectively manage a variety of complex and heterogeneous resources, thereby meeting the diverse intelligent computing acceleration needs of users.

[0028] The intelligent scene matching method of the present invention supports intelligent model deployment and orchestration processes based on scene features, and can realize intelligent matching based on scene features such as concurrency, timeliness, accuracy, and business type.

[0029] The method of the present invention can also simultaneously achieve automated deployment on matching computing resources, ultimately realizing the automated deployment of intelligent applications in the production environment;

[0030] The autonomous and controllable vertical domain intelligent computing service platform and intelligent scene matching method of the present invention can be well applied to scenarios involved in the field of artificial intelligence;

[0031] The autonomous and controllable vertical domain intelligent computing service platform and intelligent scene matching method of the present invention can be applied to scenarios such as image / video processing, text processing, voice processing, game competition, official document processing, and intelligence analysis. Attached Figure Description

[0032] Figure 1 This is a schematic diagram of the heterogeneous intelligent computing service platform structure;

[0033] Figure 2 This is a flowchart of the model development, training, and deployment process. Detailed Implementation

[0034] The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0035] This invention proposes an intelligent computing service platform based on independently controllable hardware and software, such as... Figure 1 As shown, it includes:

[0036] The heterogeneous resource scheduling engine is mainly used to provide a unified resource scheduling framework for intelligent computing service platforms. It performs resource scheduling of heterogeneous computing resources such as CPU+GPU, CPU+ASIC, and CPU+FPGA at the infrastructure level to meet the deployment requirements of intelligent applications.

[0037] The intelligent orchestration module for model training is mainly used to customize the training process of machine learning models based on scene templates. The training deployment process can be intelligently matched according to the scene template selected by the user, and multiple training deployment schemes are provided for the user to choose from. Users can also customize the data preprocessing, model training, environment configuration and other processes on the provided training deployment schemes.

[0038] The heterogeneous resource unified management module is mainly used for unified management of various types of heterogeneous resources such as CPU+GPU, CPU+ASIC, and CPU+FPGA, and supports functions such as online expansion, status monitoring, and fault alarm.

[0039] The integrated intelligent interaction front-end module is mainly used for the intelligent computing service platform to interact well with users, and to support model development, training and intelligent application inference.

[0040] The application model automatic deployment module is mainly used to package and deploy pre-trained models developed on the platform to the production environment, directly providing intelligent services to users.

[0041] Furthermore, the heterogeneous computing resources include the computing power of chips such as GPUs, FPGAs, and intelligent ASICs. It not only supports mainstream foreign heterogeneous resources, but also supports domestically developed CPUs such as Phytium and Shenwei, and domestically developed intelligent accelerator cards such as Cambricon, Bitmain, Huawei Ascend, and Baidu Kunlun. This implies that all permutations and combinations of CPUs, GPUs, FPGAs, and intelligent ASICs are included.

[0042] Furthermore, the intelligent computing framework includes not only mainstream international frameworks such as Tensorflow, PyTorch, and ONNX, but also domestic machine learning frameworks such as PaddlePaddle and Jittor.

[0043] Furthermore, the model training intelligent orchestration module allows users to select scenarios including image classification, object detection, NLP, speech recognition, speech synthesis, object recommendation, image generation, and text generation. Trainable and developable models include GhostNet, DPN, SE_ResNeXt, Xception, EfficientNet, MaskR-CNN, ShapeMask, SpineNet, yoloV3_darknet5, DeepLab_v3_s16, LSTM, ERNIE, BERT, GCN, GAT, BLSTM, WaveNet, WaveGlow, Tacotron, RNN-T, NCF, GNN, DLRM, DeepFM, NFM, MiniGo, DCGAN, PGAN, DDPG, and DQN. Available training and deployment solutions include high-performance solutions prioritizing performance and high-reliability solutions prioritizing reliability.

[0044] The present invention provides a heterogeneous intelligent computing service platform based on independently controllable software and hardware. Its deployment is mainly divided into master nodes and slave nodes. The modules deployed on the master node include: a heterogeneous resource scheduling engine 13, a comprehensive intelligent interaction front-end module 11, an application model automatic deployment module 12, and a model training intelligent orchestration module 14. The heterogeneous resource unified management module 15 is deployed on the slave nodes.

[0045] Master node workflow:

[0046] 1) Users submit deployment tasks for intelligent applications to the backend through the integrated intelligent interactive frontend;

[0047] 2) The heterogeneous resource scheduling engine 13 extracts the required resource tags and number of instances based on the deployment task information, queries the configuration database for the number of resources and resource usage under the same resource tag, forms a task instantiation scheme based on the scheduling strategy, and then distributes the instantiation scheme to the relevant slave nodes.

[0048] 3) Automatic deployment of the application model: Based on the instantiation scheme, the API of the heterogeneous resource unified management module 15 is called to create a container, and runtime components are pulled according to the container startup script and started one by one in sequence. Among them, the container image should be configured with device files in advance and include driver libraries for calling the computing resources of ASIC, GPU, FPGA and other boards. The runtime components include domestic computing frameworks such as Paddle, Jittor, TensorFlow, PyTorch, C++, Python and other dependent environments, toolkits, etc.

[0049] 4) There are two implementations of the intelligent orchestration of model training. One is that the user selects the corresponding business type, model version, runtime components, required resource type and number of instances through the comprehensive intelligent interactive front end to form a specific deployment task plan. The other is that based on the scenario-based intelligent matching algorithm, an instantiable deployment task plan is automatically formed according to the business type, timeliness target and accuracy target input by the user. This method is described in detail in Implementation Example 2.

[0050] From node workflow:

[0051] 1) After receiving the instantiation scheme distributed by the master node, the heterogeneous resource unified management module 15 extracts information such as model version and number of instances, creates containers according to the number of instances, and automatically deploys the application model. Module 12 pulls the corresponding version of the model and runtime components from the remote model repository and runtime component repository according to the container startup script and starts them.

[0052] 2) Unified Management of Heterogeneous Resources: Module 15 collects heterogeneous resource usage information using different methods depending on the managed resources. For FPGA and ASIC devices, the default Docker runtime is replaced with the custom-designed FPGA / ASIC container runtime. When an intelligent computing unit container is created, the FPGA / ASIC container runtime first executes its corresponding hook. This hook checks environment variables to determine if the container needs to use FPGA or ASIC. If so, it obtains the FPGA / ASIC device library through a device library scanner, thereby exposing the FPGA / ASIC devices for container use. Usage is determined by executing corresponding scripts to call the FPGA / ASIC manufacturer's packaged APIs to read registers or by checking function return values. For GPU devices, the determination is made by calling the board resource API interface.

[0053] 3) Unified management of heterogeneous resources: Module 15 periodically reports the resource usage of each computing node to the master node. The master node updates the resource information in the configuration database and sends alarms to users based on the reports from the slave nodes.

[0054] Users can first upload the training set for this instantiation via the Data Management 143 module, or use the platform's built-in training set. Based on the scenario's capability requirements, the priority of scenario features is determined. The Intelligent Matching 141 module intelligently matches these features to form the instantiation scheme. The Custom Orchestration 144 module provides custom configuration functionality for users to adjust the instantiation scheme. The Model Training 142 module calls the Deployment Engine 121 to instantiate and deploy the model according to the instantiation scheme and performs training. The Heterogeneous Resource Scheduling Engine 13 collects the overall hardware resource status, plans the deployment scheme, and completes resource scheduling. Users can view and analyze the model training results through the Model Development and Training UI 111. After model training, users can choose to automatically deploy the model according to the instantiation scheme or download it locally. The Model Management 122 is responsible for managing the platform's built-in model groups (image classification, object detection, NLP, speech recognition, speech synthesis, object recommendation, image generation, text generation, etc.) and the models trained on the platform. After automatic model deployment, users upload the test set via the Data Management 143 module for inference calculation. After completion, users can view the inference results through the Intelligent Application Inference UI 112. The underlying heterogeneous computing resources, such as CPU+GPU, CPU+ASIC, and CPU+FPGA, can be managed through 15 modules of unified heterogeneous resource management, enabling functions such as online management, status monitoring, and fault alarms.

[0055] This invention also relates to an intelligent scene matching method, implemented using the aforementioned autonomous and controllable intelligent computing service platform, comprising:

[0056] Users can configure the scenario features provided by the intelligent computing service platform according to the capability requirements of their desired scenarios. Available scenario features include concurrency, timeliness, accuracy, and business type. Business type refers to different business model groups, such as image classification, object detection, NLP, speech recognition, speech synthesis, object recommendation, image generation, and text generation. Concurrency refers to the number of intelligent application instances required for each inference task, defaulting to 1, which can be adjusted according to the algorithm if hardware computing power is insufficient. Timeliness refers to the user's expected computation time for this inference task; the intelligent computing service platform can automatically select the fastest processing model from the available models for deployment based on this configuration. Accuracy refers to the difference between the model's prediction result and the actual value; the platform can select from the available models based on this configuration. The user input items for this method are timeliness (speed), accuracy, and business type; the algorithm output items after intelligent matching are concurrency, hardware computing power, and the specific model.

[0057] like Figure 2As shown, this embodiment provides an intelligent scene matching method, implemented based on the autonomous and controllable vertical domain intelligent computing service platform described in Embodiment 1. The implementation of the intelligent scene matching method based on the intelligent computing service platform is described below:

[0058] Step 1: Add a new heterogeneous resource node, deploy a monitoring program and an agent program on the heterogeneous resource node, and periodically execute the monitoring program to call system commands and board drivers to obtain the resource type and usage of the heterogeneous resource node.

[0059] Step 2: Use the platform's built-in models and datasets to test the models. Execute scripts to calculate the model's inference time and accuracy. Add the unit inference time and accuracy on this resource to the model's attribute table. Repeat this process for all built-in models on the platform to obtain the unit inference time and accuracy of all built-in models on this heterogeneous resource. Establish a model-resource node mapping relationship based on the unit inference time and accuracy parameters.

[0060] Step 3: The user uploads a dataset of size D, inputs the timeliness target value T and the accuracy target value M, and selects the business type. Based on the business type label, the range of model groups for this type is determined. In this case, the unit inference time and accuracy of each model-resource node mapping within the model group are known. Further matching is then required based on the timeliness target T and the accuracy target M.

[0061] Step 4: First, remove all model-resource node mappings within the model group whose estimated inference time (estimated inference time = unit inference time * D) on all available heterogeneous resource nodes is greater than the timeliness target T or whose accuracy is less than the accuracy target M. This yields a complete set of model-resource node mappings f: {m>s}, where m is the set of models that meet the requirements, s is the set of resource nodes that meet the requirements, and f is a many-to-many mapping from m to s. There is also a case where none of the models in the model group can meet the requirement of their estimated inference time being less than the timeliness target T. In this case, calculate the concurrency P = estimated inference time / T (to ensure that at least half of the models are within the selectable range, the estimated inference time is the median of the estimated inference times of all models in the model group). Then, in this case, remove all model-resource node mappings that are greater than T / P or less than M.

[0062] Step 5: Next, based on the node resource usage information reported to the master node by the monitoring program deployed on the nodes, remove nodes in s whose resource usage does not meet the actual deployment requirements, and obtain the deployable model—resource node mapping f':{m'>s'}, where f', m', and s' are all subsets of f, m, and s, respectively.

[0063] Step Six: Finally, in the obtained model-resource node mapping f':{m'>s'}, sort them according to the accuracy parameter of each mapping relationship, and take the model-resource node mapping F with the highest accuracy as the algorithm output result.

[0064] Step 7: Based on the unit inference time t of the model-resource node mapping F, calculate the concurrency P = (D*t) / T. When P≤1, the number of instances is 1, and when P>1, the number of instances is [P]+1. Thus, the matching result is determined.

[0065] In summary, the above are merely preferred embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. An intelligent scene matching method, characterized in that, include: Step 1: Add a new heterogeneous resource node; Step 2: Use the platform's built-in models and datasets to test the model. Execute the script to calculate the time and accuracy of the model inference. Add the unit inference time and accuracy on the newly added heterogeneous resource node to the model attribute table. Repeat this process for all built-in models on the platform to obtain the unit inference time and accuracy of all built-in models on the heterogeneous resource node. Establish a mapping relationship between the model and all resource nodes based on the unit inference time and accuracy parameters. Step 3: The user uploads a dataset of size D, inputs the timeliness target value T and the accuracy target value M, and selects the business type. Then, the range of model groups for this type is determined based on the business type label. Step 4: First, remove all model-resource node mappings within the model group whose estimated inference time on all available heterogeneous resource nodes is greater than the timeliness target T or whose accuracy is less than the accuracy target M. Here, the estimated inference time = unit inference time * D. This yields all model-resource node mappings that meet the requirements, f: {m>s}, where m is the set of models that meet the requirements and s is the set of resource nodes that meet the requirements. Step 5: Next, based on the node resource usage information reported from the resource nodes to the master node, remove the resource nodes in s whose resource usage does not meet the actual deployment requirements, and obtain the actual deployable model and resource node mapping f':{m'>s'}, where f', m', and s' correspond to subsets of f, m, and s. Step Six: Finally, in the obtained model-resource node mapping f':{m'>s'}, sort them according to the accuracy parameter of each mapping relationship, and take the model-resource node mapping F with the highest accuracy as the algorithm output result; Step 7: Based on the unit inference time t of the model-resource node mapping F, calculate the concurrency P=(D*t) / T. When P≤1, the number of instances is 1, and when P>1, the number of instances is [P]+1. Thus, the result of this intelligent scene matching is determined.

2. The intelligent scene matching method as described in claim 1, characterized in that, In step one, a monitoring program and an agent program are deployed on the heterogeneous resource node. The monitoring program is executed periodically to call system commands and board drivers to obtain the resource type and usage status of the heterogeneous resource node.

3. The intelligent scene matching method as described in claim 1, characterized in that, In step four, the estimated inference time is the median of the estimated inference times of all models in the model group.

Images