Data service system construction method taking artificial intelligence computing power as core

Abstract

The invention relates to the technical field of artificial intelligence, in particular to a data service system construction method taking artificial intelligence computing power as a core, which comprises the following steps of computing power resource planning and deployment, data acquisition and preprocessing, data storage and management, artificial intelligence model development and training, data service interface design and implementation and data service monitoring. By adopting a virtualization technology, physical computing power resources are abstracted into a virtual resource pool, and computing power is dynamically allocated according to service requirements, so that idle and waste of hardware resources in a traditional mode are avoided.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and particularly relates to a data service system construction method taking artificial intelligence computing power as a core. BACKGROUND

[0002] Traditional data centralized storage is difficult to unify in a data warehouse or data lake, and thus it is difficult to realize centralized management of data.

[0003] The existing data service system construction method generally includes data collection: from different data sources such as business systems and log files, scattered data is extracted to a temporary storage area through an ETL tool. Secondly, data storage: the cleaned and converted data is loaded into a data warehouse, and is stored according to themes such as customer, product and other theme domains, and a star or snowflake model is constructed to improve query efficiency. Thirdly, data analysis: based on the data warehouse, a fixed format report is generated by using a report tool to meet the basic statistical demand; multi-dimensional analysis is performed by using an OLAP (Online Analytical Processing) tool, such as slicing, chunking and drilling, to assist decision analysis. Finally, data service: the analysis results are provided to business personnel in the form of reports and charts, or are called by other systems through an API interface, so that the data scattered in different business systems and different departments can be stored in a unified data warehouse or data lake to realize centralized management of data.

[0004] However, in the existing data service system construction method, computing power cannot be dynamically allocated according to business needs, so that hardware resources in the system are idle and wasted. SUMMARY

[0005] The present application relates to the technical field of artificial intelligence, and particularly relates to a data service system construction method taking artificial intelligence computing power as a core.

[0006] To achieve the above-mentioned purpose, the present application provides a data service system construction method taking artificial intelligence computing power as a core, comprising the following steps:

[0007] The demand analysis of artificial intelligence application scenarios is performed to determine the required computing power scale, type and performance index requirements;

[0008] According to the demand analysis results, hardware devices are selected and physically deployed to construct computing power infrastructure with high availability and scalability;

[0009] The computing power infrastructure is virtually processed to abstract physical computing power resources into a virtual computing power resource pool;

[0010] Establish a data collection interface to achieve regular data collection, clean the collected raw data, and finally standardize the cleaned data;

[0011] According to the requirements of artificial intelligence model, the preprocessing operation of feature extraction and feature selection is carried out on the data to generate feature data set suitable for model training;

[0012] Select data storage technology and establish data classification and hierarchical management system;

[0013] According to the business requirements and data characteristics, select artificial intelligence algorithm and model architecture, and use the preprocessed feature data set to train the model on the planned and deployed computing resources;

[0014] Evaluate and verify the trained model;

[0015] According to the business requirements and user usage scenarios, design data service interface, and use interface technology to realize the development and deployment of data service interface;

[0016] Establish data service monitoring system.

[0017] Among them, according to the demand analysis result, select hardware equipment and carry out physical deployment to build computing infrastructure with high availability and scalability, and the steps also include:

[0018] The hardware equipment includes servers, storage devices and network devices.

[0019] Among them, select data storage technology and establish data classification and hierarchical management system, and the steps also include:

[0020] Classify and store different types and importance of data, set corresponding access rights and security policies;

[0021] Backup important data regularly and build data index and query optimization mechanism.

[0022] Among them, according to the business requirements and data characteristics, select artificial intelligence algorithm and model architecture, and use the preprocessed feature data set to train the model on the planned and deployed computing resources, and the steps also include:

[0023] The model architecture includes deep learning model and machine learning model.

[0024] Among them, according to the business requirements and user usage scenarios, design data service interface, and use interface technology to realize the development and deployment of data service interface, and the steps also include:

[0025] The data service interface includes data query interface, model inference interface and data analysis interface.

[0026] The steps include designing a data service interface based on business requirements and user scenarios, and implementing the development and deployment of the data service interface using interface technology. The steps also include:

[0027] Use interface technology to develop and deploy data service interfaces;

[0028] Optimize the performance of data service interfaces and establish interface documentation and version management mechanisms;

[0029] Establish interface documentation and version management mechanism.

[0030] The step of establishing a data service monitoring system further includes:

[0031] Real-time monitoring of computing resource usage, data storage status, model inference performance, and interface call status;

[0032] Set monitoring indicators and alarm thresholds;

[0033] Regularly evaluate and optimize the performance of the data service system;

[0034] Establish operation and maintenance management processes and systems.

[0035] The present invention discloses a method for constructing a data service system with artificial intelligence computing power as its core. The method accurately estimates computing power scale based on business needs, selects appropriate hardware equipment and performs physical deployment, uses virtualization technology to build a virtual computing power resource pool, realizes dynamic resource allocation, accurately locates multi-source data, customizes and develops collection interfaces, deeply cleans, standardizes and features the collected data, and provides high-quality data for subsequent model training. Appropriate storage technology is selected based on data characteristics, a tiered storage strategy is implemented, data is classified and graded, and a comprehensive data backup and recovery and index query optimization mechanism is established. Appropriate algorithm models are selected based on business needs and data characteristics, distributed training technology is used to efficiently train on the planned computing power resources, and the models are evaluated, verified and optimized. Scientific and reasonable interfaces are designed to meet business needs and user usage scenarios. Appropriate technologies are used for development, and rigorous testing and performance optimization are carried out to build a comprehensive monitoring system. Through the reasonable planning and scheduling of computing power resources, resource utilization can be improved and hardware costs can be reduced. Virtualization technology is used to abstract physical computing power resources into virtual resource pools, and computing power is dynamically allocated according to business needs, thereby avoiding the idleness and waste of hardware resources in traditional models. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art.

[0037] Figure 1 This is a schematic diagram of the steps of a method for constructing a data service system with artificial intelligence computing power as the core according to the first embodiment of the present invention. DETAILED DESCRIPTION

[0038] The embodiments of the present invention are described in detail below. Examples of the embodiments are shown in the accompanying drawings. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present invention, but should not be understood as limiting the present invention.

[0039] The first embodiment of this application is:

[0040] See also Figure 1 ,in Figure 1 This is a schematic diagram of the steps of a method for constructing a data service system with artificial intelligence computing power as the core according to the first embodiment of the present invention.

[0041] The present invention provides a method for constructing a data service system with artificial intelligence computing power as the core, comprising the following steps:

[0042] S101: Conduct demand analysis for AI application scenarios to determine the required computing power scale, type, and performance requirements;

[0043] S102: Based on the demand analysis results, select hardware devices and perform physical deployment to build a computing infrastructure with high availability and scalability.

[0044] S103: Virtualize the computing infrastructure and abstract the physical computing resources into a virtual computing resource pool;

[0045] S104: Establish a data collection interface to implement regular data collection, clean the collected raw data, and finally standardize the cleaned data;

[0046] S105: Perform feature extraction and feature selection preprocessing on the data according to the requirements of the artificial intelligence model to generate a feature dataset suitable for model training;

[0047] S106: Select data storage technology and establish a data classification and hierarchical management system;

[0048] S107: Select AI algorithms and model architectures based on business needs and data characteristics, and use pre-processed feature datasets to train models on the planned computing resources.

[0049] S108: Evaluate and verify the trained model;

[0050] S109: Design data service interfaces based on business needs and user scenarios, and implement the development and deployment of data service interfaces using interface technology;

[0051] S110: Establish a data service monitoring system.

[0052] Furthermore, a comprehensive and in-depth needs analysis of AI application scenarios is fundamental to building an efficient data service system. This requires close collaboration with business departments to understand the requirements for data processing speed, model complexity, and other aspects of different business scenarios. Through a detailed needs analysis, determine the required computing power scale (floating-point operations per second, or FLOPS), type (CPU excels at logic control and general-purpose computing, GPU is suitable for massively parallel computing, and TPU is optimized for deep learning algorithms), and performance metrics (response time, throughput). Select appropriate hardware based on the needs analysis. Servers must have sufficient computing power, memory, and storage capacity to meet the needs of data processing and model training. Storage devices should consider data read and write speed, capacity, and reliability, using high-speed solid-state drives to improve data access speed. Network equipment must ensure stable data transmission and low latency. In terms of physical deployment, a reasonable rack layout and cooling design should be adopted to ensure stable hardware operation. Furthermore, a highly available and scalable computing infrastructure should be built, employing technologies such as redundant power supplies and dual-server hot standby to improve system availability. Hardware expansion ports and space should be reserved to facilitate future computing capacity expansion based on business growth. Virtualizing computing infrastructure is a key step in abstracting physical computing resources into a virtual computing resource pool. Containerization technologies (Docker) or virtual machine technologies (VMware) are used to abstract and isolate computing resources. Virtual computing resources are centrally managed and scheduled through container orchestration tools (Kubernetes) or virtual machine management platforms (OpenStack), dynamically allocating computing resources based on business needs and improving resource utilization.

[0053] Furthermore, when establishing a data collection interface, database data can be collected in real time or periodically using a database connector; for IoT device data, a message queue (Kafka) can be used for data buffering and transmission. At the same time, to ensure the stability and reliability of the collection interface and avoid data loss and errors, data cleaning is a key step in ensuring data quality. Rule-based cleaning methods can be used, such as removing non-compliant data based on data format requirements; statistical cleaning methods, such as identifying and removing outliers by calculating statistical quantities such as the mean and standard deviation; and machine learning-based cleaning methods, such as using clustering algorithms to identify and remove noisy data. The cleaned data can then be standardized to unify data formats and units. Standardization can avoid data analysis errors caused by inconsistent data formats and improve data comparability and usability. Based on the requirements of the AI ​​model, preprocessing operations such as feature extraction and selection can be performed on the data to generate a feature dataset suitable for model training.

[0054] Furthermore, select appropriate data storage technologies, such as relational databases, non-relational databases, distributed file systems, etc., and design storage architecture based on data characteristics and access requirements; establish a data classification and hierarchical management system, classify and store data of different types and importance, and set corresponding access rights and security policies; implement data backup and recovery mechanisms, regularly back up important data, and ensure data security and reliability; build data indexing and query optimization mechanisms to improve data retrieval efficiency and meet the needs of fast query and analysis.

[0055] Furthermore, according to business needs and data characteristics, select appropriate artificial intelligence algorithms and model architectures, such as deep learning models, machine learning models, etc.; use the preprocessed feature data set to train the model on the planned and deployed computing resources, and improve the accuracy and performance of the model by adjusting model parameters, optimizing training algorithms, etc.; during the model training process, adopt distributed training, parallel computing and other technologies to make full use of computing resources and shorten training time; evaluate and verify the trained model, and adopt cross-validation, test set verification and other methods to ensure the effectiveness and generalization ability of the model.

[0056] Furthermore, based on business needs and user usage scenarios, data service interfaces are designed, including data query interfaces, model inference interfaces, data analysis interfaces, etc.; appropriate interface technologies, such as RESTful API, gRPC, etc., are adopted to implement the development and deployment of data service interfaces; performance optimization of data service interfaces is performed, including interface response time optimization, concurrent processing capability optimization, etc., to ensure efficient and stable operation of the interfaces; interface documentation and version management mechanisms are established to facilitate user use and maintenance of the interfaces.

[0057] Furthermore, a data service monitoring system will be established to conduct real-time monitoring of computing resource usage, data storage status, model inference performance, interface call status, etc.; monitoring indicators and alarm thresholds will be set, and when monitoring indicators exceed the normal range, alarm information will be issued in a timely manner to notify operation and maintenance personnel to handle it; regular performance evaluation and optimization of the data service system will be conducted, and according to monitoring data and business needs, computing resource allocation will be adjusted, model parameters will be optimized, data storage structure will be improved, etc., to improve the overall performance and quality of data services; operation and maintenance management processes and systems will be established to standardize the management of fault handling, version upgrades, security maintenance, etc. of the data service system.

[0058] When using this method for building a data service system with artificial intelligence computing power as its core, the computing power scale is accurately estimated based on business needs, appropriate hardware equipment is selected and physically deployed, virtualization technology is used to build a virtual computing power resource pool, resource dynamic allocation is achieved, multi-source data is accurately located, and custom collection interfaces are developed. The collected data is deeply cleaned, standardized, and feature-engineered to provide high-quality data for subsequent model training. Appropriate storage technology is selected based on data characteristics, a tiered storage strategy is implemented, data is classified and graded, and a comprehensive data backup and recovery and index query optimization mechanism is established. Appropriate algorithm models are selected based on business needs and data characteristics, distributed training technology is used to efficiently train on the planned computing power resources, and the models are evaluated, verified, and optimized. Scientific and reasonable interfaces are designed to meet business needs and user scenarios. Appropriate technologies are used for development, rigorous testing and performance optimization are carried out, and a comprehensive monitoring system is established. Through the reasonable planning and scheduling of computing power resources, resource utilization can be improved and hardware costs can be reduced. Therefore, virtualization technology is used to abstract physical computing power resources into virtual resource pools, and computing power is dynamically allocated according to business needs, thus avoiding the idleness and waste of hardware resources in the traditional model.

[0059] The above disclosure is merely one or more preferred embodiments of the present application and is not intended to limit the scope of the present application. A person skilled in the art will understand that all or part of the processes of the above embodiments and equivalent changes made in accordance with the claims of the present application are still within the scope of the present application.

Claims

1. A method for constructing a data service system with artificial intelligence computing power as the core, characterized in that: The following steps are involved: Conduct demand analysis for AI application scenarios to determine the required computing power scale, type, and performance requirements; Based on the demand analysis results, select hardware devices and perform physical deployment to build a computing infrastructure with high availability and scalability; Virtualize the computing infrastructure and abstract physical computing resources into a virtual computing resource pool; Establish a data collection interface to achieve regular data collection, clean the collected raw data, and finally standardize the cleaned data; According to the requirements of the artificial intelligence model, preprocess the data with feature extraction and feature selection to generate a feature data set suitable for model training; Select data storage technology and establish a data classification and hierarchical management system; Select AI algorithms and model architectures based on business needs and data characteristics, and use pre-processed feature datasets to train models on planned and deployed computing resources. Evaluate and validate the trained model; Design data service interfaces based on business needs and user scenarios, and implement the development and deployment of data service interfaces using interface technology; Establish a data service monitoring system.

2. The method for constructing a data service system with artificial intelligence computing power as the core according to claim 1, characterized in that: Based on the demand analysis results, select hardware devices and perform physical deployment to build a highly available and scalable computing infrastructure. This step also includes: The hardware devices include servers, storage devices and network devices.

3. The method for constructing a data service system with artificial intelligence computing power as the core according to claim 1, characterized in that: Selecting data storage technology and establishing a data classification and hierarchical management system, the steps also include: Categorize, store, and manage data of different types and importance, and set corresponding access rights and security policies; Back up important data regularly and build data indexing and query optimization mechanisms.

4. The method for constructing a data service system with artificial intelligence computing power as the core according to claim 1, characterized in that: Based on business needs and data characteristics, select an AI algorithm and model architecture, and use the preprocessed feature dataset to train the model on the planned computing resources. This step also includes: The model architecture includes a deep learning model and a machine learning model.

5. The method for constructing a data service system with artificial intelligence computing power as the core according to claim 1, characterized in that: Design data service interfaces based on business requirements and user scenarios, and implement the development and deployment of data service interfaces using interface technology. This step also includes: The data service interface includes a data query interface, a model reasoning interface and a data analysis interface.

6. The method for constructing a data service system with artificial intelligence computing power as the core according to claim 5, characterized in that: Design a data service interface based on business requirements and user scenarios, and implement the development and deployment of the data service interface using interface technology. The steps also include: Use interface technology to develop and deploy data service interfaces; Optimize the performance of data service interfaces and establish interface documentation and version management mechanisms; Establish interface documentation and version management mechanism.

7. The method for constructing a data service system with artificial intelligence computing power as the core according to claim 1, characterized in that: Establishing a data service monitoring system, the steps also include: Real-time monitoring of computing resource usage, data storage status, model inference performance, and interface call status; Set monitoring indicators and alarm thresholds; Regularly evaluate and optimize the performance of the data service system; Establish operation and maintenance management processes and systems.

Images