An application scenario-oriented domestication system evaluation method

Abstract

The application discloses a kind of application scene-oriented localization system evaluation method.The method is: 1) general service architecture for application system is constructed, including: traffic processing service module, data forwarding service module, data processing service module, data storage service module, virtualization platform service module, cloud platform service module;2) a business system is selected and divided into multiple service modules;Business system is deployed in multiple servers, and each server runs localization operating system, at least one service module is run in the localization operating system;3) each module in general service architecture respectively obtains the index value when the service module is run in the localization operating system of each server;4) according to the index value of each service module and the index threshold value of corresponding service module, a comprehensive evaluation value is obtained according to the comparison result of each index value, and the evaluation result of localization operating system is determined according to the comprehensive evaluation value.

Description

Technical Field

[0001] This invention belongs to the field of Internet information technology, and specifically relates to a method for evaluating domestically produced systems for application scenarios. Background Technology

[0002] Promoting the comprehensive application of domestically developed systems is a strategic requirement for achieving independent control over key technologies in fields such as finance, e-government, and network communications. For institutions undertaking major national projects, such as aerospace and aviation, the proportion of domestically produced equipment used will gradually increase. However, the compatibility, stability, functionality, performance, and post-launch operational effectiveness of typical application business systems on domestic platforms still need to be evaluated.

[0003] In engineering practice, there are two main reasons hindering the large-scale use of domestically developed platforms. First, the replacement of older systems with domestically developed platforms is still in its early stages and has not yet achieved widespread application. Directly porting a wealth of software to domestic platforms carries potential risks, such as compatibility issues before operation and problems with functionality and stability after operation. In particular, there is a lack of practical data to support the compatibility, stability, functionality, and performance of typical application systems running on domestically developed platforms, thus failing to provide a basis for platform selection. Second, after application systems are running on domestically developed platforms, the monitoring indicator system related to application system operation and maintenance needs to be rebuilt, and relevant monitoring indicators need to be added, deleted, or adjusted based on the monitoring results of domestically developed platforms.

[0004] Existing technologies cannot meet the evaluation requirements of domestically produced systems for application scenarios.

[0005] 1) Existing technologies mainly evaluate those based on general-purpose processors (X86, ARM) and operating systems (Windows, Linux), and lack effective evaluation of those based on domestically developed platforms.

[0006] 2) Existing system evaluation technologies generally suffer from a single measurement perspective, evaluating only from the perspective of functional or performance attributes, and failing to provide a comprehensive evaluation of different stages throughout the system's lifecycle.

[0007] 3) Existing assessment technologies evaluate system indicators of the platform, but their assessment granularity is relatively coarse, making it difficult for users to perceive accurately and failing to effectively support the operation and maintenance of application business systems.

[0008] 4) Existing system evaluation strategies lack comprehensive consideration from multiple dimensions, making it difficult to form an overall perception capability based on business scenarios. Summary of the Invention

[0009] The purpose of this invention is to overcome the shortcomings of the prior art and provide a domestic system evaluation method for application scenarios, so as to solve the problem that the prior art cannot achieve the adaptation of the application system before going online and the operation evaluation after going online based on the domestic platform environment.

[0010] To achieve the above objectives, the technical solution of the present invention is as follows:

[0011] A method for evaluating domestically developed systems for application scenarios, comprising the following steps:

[0012] 1) Construct a general business architecture for the application system; the general business architecture includes: traffic processing business module, data forwarding business module, data processing business module, data storage business module, virtualization platform business module, and cloud platform business module;

[0013] 2) Select a business system and divide it into multiple business modules; the business system is deployed on multiple servers, each server runs a domestically produced operating system, and at least one of the business modules runs in the domestically produced operating system;

[0014] 3) Each module in the general business architecture obtains the indicator values ​​when the business module is running on the domestic operating system of each server;

[0015] 4) Compare the indicator value of each business module obtained in step 3) with the indicator threshold of the corresponding business module, obtain a comprehensive evaluation value based on the comparison results of each indicator value, and determine the evaluation result of the domestic operating system based on the comprehensive evaluation value.

[0016] Furthermore, according to Calculate the indicator values ​​for the business module, and then according to... Calculate the comprehensive evaluation value S; M of the business system. t Let M0 represent the metric value of the t-th business module, M0 represent the initial evaluation value of the t-th business module, a1 represent the weight of the adaptability evaluation in the evaluation of the t-th business module, n1 represent the number of adaptability evaluation metrics under the t-th business module, and w represent the metric value of the t-th business module. i N represents the deduction weight for anomalies in the i-th suitability assessment metric. i denoted by , a2 represents the number of times the i-th adaptability indicator exceeds the threshold; a2 represents the weight of the software commonality evaluation indicator in the evaluation of the t-th business module; n2 represents the number of software commonality evaluation operation indicators under the t-th business module; w j N represents the deduction weight for the j-th software common evaluation index anomaly. jdenoted as , indicating the number of times the j-th software commonality assessment operation indicator exceeds the threshold and triggers an abnormal warning; a3 represents the weight of the application business focus assessment in the t-th business module assessment; n3 represents the number of application business focus assessment operation indicators under the t-th business module; w k N represents the deduction weight for anomalies in the key evaluation indicators of the k-th application business. k This represents the number of times the key evaluation indicator for the k-th application business exceeds the threshold and triggers an abnormal warning; m is the number of the business modules, and sw t Let be the weight coefficient of the t-th business module.

[0017] Furthermore, the compatibility evaluation indicators for the traffic processing service module when calculating the t-th service module include the version of the packet capture driver adapted to the domestic platform, the packet capture network card model, and the number of packet capture network ports; the common software evaluation indicators for the traffic processing service module when calculating the t-th service module include device manufacturer information, processor model, number of processor cores, total device memory size, device hard disk storage size, operating system version, CPU usage of the packet capture driver, CPU usage of the software, traffic volume of each network port, device load, hard disk read / write IO, device hard disk utilization, device memory usage, packet capture driver running status and restart status, packet loss count of the packet capture driver, traffic received by the traffic processing software, packet loss of the traffic processing software, memory consumption of the traffic processing software, single-core CPU usage of the traffic processing software, and service log content; the key application service evaluation indicators for the traffic processing service module when calculating the t-th service module include the packet capture driver adaptation version, packet capture network card model, number of packet capture network ports, packet loss count of the packet capture driver, traffic received by the traffic processing software, and packet loss of the traffic processing software.

[0018] Furthermore, the adaptation evaluation indicators for the data forwarding service module when calculating the t-th service module include the forwarding network card model, application software adaptation version, and the status of the resident port of the forwarding software; the common software evaluation indicators for the data forwarding service module when calculating the t-th service module include equipment manufacturer information, processor model, number of processor cores, total device memory size, device hard disk storage size, operating system version, operating system kernel version, device load, hard disk I / O read / write speed, device hard disk utilization, overall memory usage, CPU usage, forwarding network card bandwidth, forwarding network card packet loss, forwarding software CPU usage percentage, forwarding software throughput, and service operation response time; the key application service evaluation indicators for the data forwarding service module when calculating the t-th service module include the forwarding network card model, forwarding network card transmission and reception traffic, forwarding network card packet loss, the status of the resident port of the forwarding software, the forwarding software throughput, and the service operation response time.

[0019] Furthermore, the adaptation evaluation indicators for the data processing service module when calculating the t-th service module include the service data network card model, application software adaptation version, application software running status, and application software resident port status; the common software evaluation indicators for the data processing service module when calculating the t-th service module include equipment manufacturer information, processor model, number of processor cores, total device memory size, device hard disk storage size, operating system version, operating system kernel version, device load, hard disk I / O read / write speed, device hard disk utilization, overall memory usage, CPU usage, service operation log information, service data network card packet loss, service data network card bandwidth, application software memory consumption, application software CPU usage, and service response time; the key application service evaluation indicators for the data processing service module when calculating the t-th service module include service operation log information, service data network card packet loss, and service operation response time.

[0020] Furthermore, the adaptation evaluation indicators for the data storage service module when calculating the t-th service module include the service data network card model, storage software running status, application software resident port status, storage software stress test status, and storage software performance benchmark test status. The common software evaluation indicators for the data storage service module when calculating the t-th service module include operational evaluation indicators such as device manufacturer information, processor model, number of processor cores, total device memory size, device hard disk storage size, operating system version, operating system kernel version, device load, hard disk I / O read / write speed, overall memory usage, CPU usage, service data network card bandwidth, service data network card traffic volume and packet loss, storage software memory consumption, storage software CPU usage, storage software data throughput, and storage software stress test and performance status. The key application service evaluation indicators for the data storage service module when calculating the t-th service module include the application software resident port status, storage software stress test status, and storage software performance benchmark test status.

[0021] Furthermore, the adaptation evaluation indicators for the virtualization platform business module when calculating the t-th business module include the virtualization software version and the virtualization software running status; the common software evaluation indicators for the virtualization platform business module when calculating the t-th business module include equipment manufacturer information, processor model, number of processor cores, total device memory size, device hard disk storage size, operating system version, operating system kernel version, device load, hard disk I / O read / write speed, device hard disk utilization, overall memory usage, CPU usage, virtualization network bandwidth, virtualization platform software CPU usage and scheduling, software memory usage and scheduling, and software running status; the key application business evaluation indicators for the virtualization platform business module when calculating the t-th business module include virtualization network bandwidth and virtualization platform software CPU and memory usage and scheduling.

[0022] Furthermore, the adaptation evaluation indicators for the cloud platform business module when calculating the t-th business module include cloud service architecture adaptation and cloud service architecture software operation status; the common software evaluation indicators for the cloud platform business module when calculating the t-th business module include device manufacturer information, processor model, number of processor cores, total device memory size, device hard disk storage size, operating system version, operating system kernel version, device load, hard disk IO read / write speed, device hard disk utilization, CPU usage, cloud platform system CPU usage, cloud platform system memory usage, cloud platform network bandwidth, and cloud platform data interaction volume; the key application business evaluation indicators for the cloud platform business module when calculating the t-th business module include cloud platform CPU and memory scheduling, data interaction volume, and platform network bandwidth.

[0023] Furthermore, the method for determining the threshold values ​​for each indicator in each business module is as follows: For the t-th business module, firstly, obtain the historical indicator data for each indicator in the t-th business module, and divide the historical indicator data for the indicator according to whether it is a working day or not, to obtain historical indicator data with the attribute of working day and historical indicator data with the attribute of non-working day; exclude abnormal data in the corresponding historical indicator data based on the historical indicator data of working day, and exclude abnormal data in the corresponding historical indicator data based on the historical indicator data of non-working day; then calculate the average value M of the historical data for the indicator based on the historical indicator data after excluding abnormal data; then set the threshold range of the indicator for the t-th business module to (1-α)M, (1+α)M, where α is the set offset value.

[0024] The domestic system evaluation method for application scenarios in this invention mainly involves the following:

[0025] 1) General business architecture of typical application systems on the abstract domestic platform

[0026] Unlike the operating system-level architectures proposed by Windows and Linux, this invention abstracts a general business architecture for application systems. The evaluation object of this invention is the business system of a large organization with numerous servers, each running a domestically developed operating system. The ported business system runs on each server's domestically developed operating system. The domestically developed operating system on the organization's servers is evaluated by using various metrics from the ported business system running on this large organization.

[0027] Currently, numerous application business systems based on domestically developed platforms exist within organizations or institutions, with server scales ranging from hundreds to tens of thousands. In typical systems, the general business architecture generally consists of several business modules: traffic processing, data forwarding, data processing, data storage, virtualization platform, and cloud platform. The application system framework of this invention is as follows: Figure 1 As shown:

[0028] Traffic processing business module: The front-end cluster processes and analyzes the traffic transmitted by network devices, mainly focusing on the reception and processing of raw network traffic;

[0029] Data processing business module: includes coarse assembly cluster, analysis and processing cluster, dynamic configuration generation mechanism, data aggregation mechanism, etc., focusing on data integrity and content analysis;

[0030] Data forwarding service module: includes configuration distribution mechanism and dynamic configuration distribution mechanism, mainly used to transmit system business rules;

[0031] Data storage service module: involves data storage clusters, data storage centers, and configuration libraries;

[0032] Virtualization platform business module: Currently, it is used less in typical application business systems and is often used in integrated office scenarios. With the popularization of resource conservation, unified desktop management and virtualization concepts, it will also involve the use of domestic products. The main focus is on the use of virtual platform office software under domestic equipment and the utilization of resources.

[0033] Cloud platform business module: Similar to the virtualization platform business module, it is currently used less in typical application business systems. However, with the widespread use of cloud platforms in life, it will also involve the use of domestic products. The main focus is on the operation of various cloud platform services and resource utilization under domestic equipment.

[0034] 2) Lifecycle-based assessment

[0035] ① Compatibility Assessment: This mainly involves hardware-related assessments to ensure that application software or data is portable and transferable on domestic platforms. The assessment evaluates the compatibility of application software or data on mainstream domestic platforms, such as Loongson, HiSilicon, Phytium, and Zhaoxin. Specific assessment content focuses on: device manufacturer information, processor model, number of processor cores, device memory, and device hard drive specifications.

[0036] ② Operational assessment: This involves the status of various indicators after the application software is running. The operational assessment is divided into common software assessment and application business focus assessment.

[0037] • Common software assessment: An assessment of the general operating status of application software on a domestic platform, focusing on processor, memory, hard disk utilization, operating system, device load, etc.

[0038] • Application Business Focus Assessment: For different application software running on the domestic platform, an assessment is conducted based on key business indicators. The assessment focuses on: business operation response time, application software resident port status, storage software CPU usage, storage software memory consumption, etc.

[0039] 3) Fine-grained evaluation based on business scenarios

[0040] By combining each business scenario with fine-grained metrics, an evaluation vector is obtained for each evaluation dimension.

[0041] The evaluation vector for the traffic processing service module consists of three parts: adaptation evaluation indicators, common software evaluation indicators, and key traffic processing service evaluation indicators. The key traffic processing service evaluation indicators primarily reflect the reception and processing of raw network traffic. For the traffic processing service module, the adaptation evaluation indicators are based on monitoring and collecting the runtime status values ​​of stream processing software, such as a fast packet processing application based on DPDK, on ​​a domestic platform. Application software resource consumption is a common evaluation indicator, while packet throughput and packet loss, i.e., packets processed per second and packet loss due to performance issues of the domestic platform, are the key evaluation indicators for the traffic processing service module.

[0042] The evaluation vector for the data forwarding service module consists of three parts: adaptation evaluation indicators, common software evaluation indicators, and key data forwarding service evaluation indicators. The key data forwarding service evaluation indicators focus on business data interaction, emphasizing the forwarding network interface card (NIC) transmit / receive traffic and packet loss. For the data forwarding service module, such as a forwarding program running on a domestic platform that consumes data from a Kafka cluster to a data center repository, its operational status is monitored or collected as an adaptation evaluation indicator. The system load and CPU / MEM resource consumption of the domestic platform are collected as common software evaluation indicators. The data throughput and data loss of the forwarding network interface are the key forwarding service evaluation indicators.

[0043] The evaluation vector for the data processing business module consists of three parts: adaptation evaluation indicators, common software evaluation indicators, and key data processing business evaluation indicators. Among these, the key data processing business evaluation indicators focus on reflecting data integrity and content analysis. For a data processing business module, such as a data stream splicing and analysis business system software, its operating status on a domestic platform is the adaptation evaluation indicator; the load on the domestic system and the consumption of CPU and MEM resources during software operation are the common software evaluation indicators; and the integrity of data splicing and the concurrent data analysis capacity that the software can handle are the key data processing business evaluation indicators.

[0044] The evaluation vector for the data storage service module consists of three parts: adaptation evaluation indicators, common software evaluation indicators, and key evaluation indicators for the data storage service. The key evaluation indicators for the data storage service include the storage software adaptation version, software CPU and MEM scheduling, and stress and performance testing indicators. For data storage service modules, such as cluster architecture libraries like HIVE and ClickHouse databases, whether they can run normally on domestic platforms, or the version that runs normally, constitutes the data storage service module adaptation evaluation indicator. The load on the domestic platform and CPU single-core usage are common software evaluation indicators. Performance indicators such as cluster data synchronization, concurrent connections, and single-connection data throughput are key evaluation indicators for the data storage service.

[0045] The evaluation vector for the virtualization platform business module consists of three parts: adaptation evaluation indicators, common software evaluation indicators, and key virtualization platform business evaluation indicators. The key virtualization platform business evaluation indicators primarily focus on the virtualization software adaptation version and software CPU and MEM scheduling metrics. The operation of the virtualization platform on domestically produced equipment, as well as the usage of commonly used software such as Windows Office, serve as adaptation evaluation indicators. The virtualization platform's CPU and MEM resource consumption, disk read / write speed, and network card bandwidth are common evaluation indicators, and also key virtualization platform business evaluation indicators.

[0046] The cloud platform business module evaluation vector consists of three parts: adaptation evaluation indicators, common software evaluation indicators, and key cloud platform business evaluation indicators. The key cloud platform business evaluation indicators mainly involve CPU and MEM scheduling, throughput, and service operation indicators. The operation of the cloud platform business module on domestically produced equipment serves as the adaptation evaluation indicator, while network interface card bandwidth throughput, CPU and MEM resource consumption, and disk I / O read / write speed are common evaluation indicators, and also key cloud platform business evaluation indicators.

[0047] 4) Dynamically establish indicator thresholds based on historical data

[0048] Based on the characteristics of each business, analyze historical data of indicators to obtain dynamic thresholds for the indicators, distinguish between working days and non-working days, and exclude abnormal historical indicator data to obtain highly accurate dynamic thresholds. This avoids the negative impacts caused by fixed thresholds that are too large or too small, such as insensitive response and frequent warnings.

[0049] 5) Comprehensive evaluation of multi-dimensional indicators

[0050] Further analysis of the indicator data collected from the business system, data preprocessing, and completion of multi-dimensional evaluation and grading of the business system's adaptation and operation on the domestic platform will improve the overall perception of the business system by operation and maintenance personnel.

[0051] 6) Monitor the application system in real time and issue automatic warnings.

[0052] Establish a real-time monitoring system to display the status of indicators for each business module in real time, and use the monitoring module to monitor the values ​​of evaluation indicators. Based on the comprehensive evaluation and grading results of multi-dimensional indicators, the monitoring module will issue an early warning.

[0053] Compared with the prior art, the positive effects of the present invention are as follows:

[0054] This invention establishes a domestic system evaluation system oriented towards application scenarios, including an adaptation system before application system launch and an operational evaluation system after launch; this invention provides a basis for selecting domestic platforms and can support a smooth transition of operation and maintenance work to domestic platforms. Attached Figure Description

[0055] Figure 1 This is a diagram of the application business system of the present invention.

[0056] Figure 2 This is a diagram of the overall architecture of the present invention. Detailed Implementation

[0057] The present invention will now be described in further detail with reference to the accompanying drawings. The examples given are only for explaining the present invention and are not intended to limit the scope of the present invention.

[0058] This invention provides a method for evaluating domestically produced systems oriented towards application scenarios. The specific evaluation scheme is as follows:

[0059] (1) Obtain the general business architecture of typical application systems

[0060] First, a target application system (such as a typical application system) running on a large organization or institution is divided into business modules to allow for the design and evaluation of metrics for each module. Organizations or institutions typically have numerous existing application systems, with server scale ranging from hundreds to tens of thousands. In a typical system, the general technical architecture generally consists of several business modules: traffic processing, data forwarding, data processing, data storage, virtualization platform, and cloud platform. A typical application system framework is as follows: Figure 1 As shown:

[0061] Traffic processing business module: The front-end cluster processes and analyzes the traffic transmitted by network devices, mainly focusing on the reception and processing of raw network traffic;

[0062] Data processing business module: includes coarse assembly cluster, analysis and processing cluster, dynamic configuration generation mechanism, data aggregation mechanism, etc., focusing on data integrity and content analysis;

[0063] Data forwarding service module: includes configuration distribution mechanism and dynamic configuration distribution mechanism, mainly used to transmit system business rules;

[0064] Data storage service module: involves data storage clusters, data storage centers, and configuration libraries;

[0065] Typical application systems do not involve virtualization platforms and cloud platforms, but with the widespread use of virtualization and cloud platforms in daily life, the use of domestically produced platforms will be required. Therefore, the business modules for virtualization platforms and cloud platforms will be added.

[0066] (2) Determine the evaluation content of the business modules

[0067] For multiple business modules of the application system, an evaluation was conducted from the perspectives of adaptability and operation, defining adaptability evaluation and operational evaluation indicators. Furthermore, common software evaluation content and key application business evaluation content were defined. The specific evaluation content for each business module is as follows:

[0068] ① Compatibility Assessment: This mainly involves hardware compatibility assessment to ensure that application software or data is portable and transferable on domestic platforms. It includes compatibility testing of application software or data on mainstream domestic platforms, such as Loongson, HiSilicon, Phytium, and Zhaoxin. This includes information such as equipment manufacturer, compatible hardware models, and compatible software versions.

[0069] ② Operational evaluation: This involves the status of various indicators after the application software is running.

[0070] • Common Software Assessment: This assesses the operational status of application software on various general-purpose platforms. Relevant indicators include processor model, number of processor cores, device memory size, device hard disk storage size, hard disk I / O activity, hard disk utilization, operating system version, and device load.

[0071] • Application Business Focus Assessment: An assessment is conducted based on key business metrics for different application software running on the domestic platform. These metrics include business response time, application software resident port status, storage software CPU usage, and storage software memory consumption.

[0072] (3) Perform the steps for each business module to obtain the evaluation vector for each evaluation dimension.

[0073] The overall framework of the present invention is as follows Figure 2 As shown, a typical application system consists of six business modules: traffic processing, data forwarding, data processing, data storage, virtualization platform, and cloud platform. The evaluation of each module includes adaptation and operational evaluation. The operational evaluation is further divided into common software evaluation content and key application business evaluation content. The evaluation indicators for each module are designed as follows:

[0074] 1) Traffic Processing Business Module: The domestic platform is used in the traffic processing business system. Adaptation evaluation indicators include the version of the packet capture driver compatible with the domestic platform, the packet capture network card model, and the number of packet capture network ports. Common software evaluation indicators during operation include: device manufacturer information, processor model, number of processor cores (physical CPU and logical CPU), total device memory size, device hard disk storage size, operating system version, CPU usage of the packet capture driver, CPU usage of the software, traffic volume (packets and bits) for each network port, device load, hard disk I / O, device hard disk utilization, device memory usage, packet capture driver running status and restart status, packet loss count of the packet capture driver, traffic received by the traffic processing software (packets and bits), packet loss status of the traffic processing software, memory consumption of the traffic processing software, single-core CPU usage of the traffic processing software, and business log content. Key evaluation indicators for application services include packet capture driver adaptation version, packet capture network card model, number of packet capture network ports, number of packet loss by the packet capture driver, traffic received by the traffic processing software (number of packets and bits), and packet loss by the traffic processing software; in this invention, "device" refers to a server running the business system.

[0075] 2) Data Forwarding Service Module: Adaptation evaluation indicators include forwarding network interface card (NIC) model, application software adaptation version, and the status of the resident ports of the forwarding software. Common software evaluation indicators during operation include device manufacturer information, processor model, number of processor cores (physical CPU and logical CPU), total device memory size, device hard disk storage size, operating system version, operating system kernel version, device load status, hard disk I / O read / write speed, device hard disk utilization, overall memory usage, CPU usage, forwarding NIC bandwidth, forwarding NIC packet loss, forwarding software CPU usage percentage, forwarding software throughput, and service operation response time. Key evaluation indicators for application services include forwarding NIC model, forwarding NIC transmit / receive traffic, forwarding NIC packet loss, the status of the resident ports of the forwarding software, forwarding software throughput, and service operation response time.

[0076] 3) Data Processing Service Module: The data processing service module's capabilities are comprehensively evaluated based on indicators such as the operation, performance, and compatibility of the business data network interface card (NIC) and data processing software. Compatibility evaluation indicators include the NIC model, application software compatibility version, application software running status, and application software resident port status. Common software evaluation indicators in the operation assessment include equipment manufacturer information, processor model, number of processor cores (physical CPU and logical CPU), total device memory size, device hard disk storage size, operating system version, operating system kernel version, device load status, hard disk I / O read / write speed, device hard disk utilization, overall memory usage, CPU usage, service operation log information, NIC packet loss / reception status, NIC bandwidth status, application software memory consumption, application software CPU usage, and service response time. Key evaluation indicators for the application service include service operation log information, NIC packet loss / reception status, and service operation response time.

[0077] 4) Data Storage Service Module: This module primarily focuses on storage software compatibility, CPU and memory scheduling, and stress and performance testing metrics. Compatibility evaluation metrics include the business data network interface card (NIC) model, storage software operating status, application software resident port status, storage software stress testing results, and storage software performance benchmark testing results. Common software evaluation metrics in the operational evaluation include device manufacturer information, processor model, number of processor cores (physical CPU and logical CPU), total device memory size, device hard disk storage size, operating system version, operating system kernel version, device load status, hard disk I / O read / write speed, overall memory usage, CPU usage, business data NIC bandwidth, business data NIC traffic volume and packet loss, storage software memory consumption, storage software CPU usage, storage software data throughput, and storage software stress testing and performance. Key evaluation metrics for application services include application software resident port status, storage software stress testing results, and storage software performance benchmark testing results.

[0078] 5) Virtualization Platform Business Module: This module primarily focuses on virtualization software version compatibility and software CPU and memory scheduling metrics. Compatibility evaluation metrics include virtualization software version and virtualization software running status. Common software evaluation metrics in the operational evaluation include device vendor information, processor model, number of processor cores (physical CPU and logical CPU), total device memory size, device hard disk storage size, operating system version, operating system kernel version, device load status, hard disk I / O read / write speed, device hard disk utilization, overall memory usage, CPU usage, virtualization network bandwidth, virtualization platform software CPU usage and scheduling, software memory usage and scheduling, and software running status. Key evaluation metrics for application services include virtualization network bandwidth and virtualization platform software CPU and memory usage and scheduling.

[0079] 6) Cloud Platform Business Module: This mainly involves CPU and memory scheduling, throughput, and service operation metrics. Adaptation evaluation metrics include cloud service architecture adaptation and cloud service architecture software operation status. Common software metrics in the operation evaluation include device vendor information, processor model, number of processor cores (physical CPU and logical CPU), total device memory size, device hard disk storage size, operating system version, operating system kernel version, device load status, hard disk I / O read / write speed, device hard disk utilization, CPU usage, cloud platform system CPU usage, cloud platform system memory usage, cloud platform network bandwidth, and cloud platform data exchange volume. Key evaluation metrics for application services include cloud platform CPU and memory scheduling, data exchange volume, and platform network bandwidth.

[0080] The specific testing methods for evaluating each business module and obtaining the evaluation results of each indicator for the business module are as follows:

[0081] The shell script is executed, and the script executes different shell commands according to each different evaluation metric to automatically obtain the metric collection information.

[0082] (4) Dynamically establish indicator thresholds based on historical data

[0083] Establishing static thresholds relies on manual monitoring and setting of thresholds by operations and maintenance personnel. However, the thresholds for most evaluation indicators across different business modules vary over time, and it's difficult to manually set accurate thresholds manually. This approach uses historical dynamic data from indicators analyzed based on business characteristics to obtain dynamic thresholds for the indicators. It differentiates between weekdays and non-weekdays, while excluding abnormal historical indicator data, resulting in more accurate dynamic thresholds. This avoids the negative impacts of fixed thresholds being too large or too small, such as sluggish response and frequent alerts. The specific steps are as follows:

[0084] 1) If it is a workday, retrieve the historical indicator data for the corresponding time period with the workday attribute. If it is a non-workday, retrieve the historical indicator data for the corresponding time period with the non-workday attribute.

[0085] 2) Exclude abnormal data from the corresponding historical indicator data.

[0086] 3) Take the average value M of the historical data after excluding abnormal historical indicator data, and set the offset value α to further calculate the upper and lower thresholds (1-α)M and (1+α)M.

[0087] (5) Comprehensive evaluation of multi-dimensional indicators

[0088] Further analysis of the indicator data collected from the business system, followed by data preprocessing, is conducted to complete a multi-dimensional evaluation and scoring of the business system on the domestic platform, thereby improving the overall perception of the business system among operations and maintenance personnel. The specific steps are as follows:

[0089] 1) First, the collected data is preprocessed. Since some data is missing, for missing indicators, K-nearest neighbors are used to impute the missing values. This involves selecting the K nearest neighbors from the complete historical samples that are closest to the missing sample, and using the weighted average of the observed values ​​in the complete samples as the imputed value. The advantages are high accuracy and simple implementation.

[0090] 2) By combining indicator thresholds and automatic early warnings from the monitoring system, abnormal indicator data is obtained. Furthermore, using pre-defined evaluation weights based on expert knowledge, the overall operational suitability of the business system is assessed through a multi-dimensional evaluation model.

[0091] The evaluation score of the business module on the domestic platform is calculated according to the following formula:

[0092]

[0093] M t Let represent the evaluation score of the t-th business module on the domestic platform, M0 represent the initial evaluation score of the business module, a1 represent the weight of the adaptability evaluation in the overall evaluation of the business module, n1 represent the number of adaptability evaluation indicators under the business module, and w i N represents the deduction weight for anomalies in the i-th suitability assessment metric. i denoted by , a2 represents the number of times the i-th adaptability indicator exceeds the threshold; a2 represents the weight of the software commonality operation assessment in the assessment of this business module; n2 represents the number of software commonality operation assessment indicators under the business module; w j N represents the deduction weight for the j-th common software operation evaluation index anomaly. j This indicates the number of times the j-th common software evaluation performance indicator exceeds the threshold, resulting in an anomaly warning; a3 represents the weight of the application business focus evaluation in the evaluation of this business module; n3 represents the number of application business focus evaluation performance indicators under the business module; w k N represents the deduction weight for anomalies in the key evaluation indicators of the k-th application business. k This indicates the number of times the key evaluation indicator for the k-th application business exceeds the threshold, resulting in an abnormal warning.

[0094] Then, the multi-dimensional evaluation score of the business system on the domestic platform is calculated according to the following formula:

[0095]

[0096] S represents the overall score of the business system on the domestic platform across multiple dimensions, m is the number of modules in the business system, and sw t These are the evaluation weighting coefficients for the sub-modules of the business system.

[0097] When M0 is 100, the control weight coefficient ensures that the multi-dimensional evaluation score S of the business system on the domestic platform remains between 0 and 100. If S ≥ 90, the business system's operation on the domestic platform is excellent, the performance of each business module is stable, and the domestic platform is well adapted to the business system; if 80 ≤ S < 90, the business system's adaptation and operation on the domestic platform is good; if 60 ≤ S < 80, the business system's adaptation and operation on the domestic platform is moderate; if S < 60, the business system's adaptation and operation on the domestic platform is poor, requiring inspection of the indicator data of each business module and analysis of the causes of the anomalies.

[0098] (6) Monitor application systems in real time and issue automatic warnings

[0099] The measured evaluation index values ​​of the application system are input into the monitoring module in real time. First, it is determined whether the evaluation result meets the excellent level. If the condition is met, the conclusion is output that the application system is normal; if the condition is not met, the conclusion is output and the monitoring module issues an early warning.

[0100] Although specific embodiments of the invention have been disclosed for illustrative purposes to aid in understanding and implementing the invention, those skilled in the art will understand that various substitutions, variations, and modifications are possible without departing from the spirit and scope of the invention and the appended claims. Therefore, the invention should not be limited to the content disclosed in the preferred embodiments, and the scope of protection claimed by the invention is defined by the claims.

Claims

1. An application scenario-oriented localization system evaluation method, characterized in that, The steps of the method include: 1) Construct a general business architecture for the application system; the general business architecture includes: traffic processing business module, data forwarding business module, data processing business module, data storage business module, virtualization platform business module, and cloud platform business module; 2) Select a business system and divide it into multiple business modules; the business system is deployed on multiple servers, each server runs a domestically produced operating system, and at least one of the business modules runs in the domestically produced operating system; 3) Each module in the general business architecture obtains the indicator values ​​when the business module is running on the domestic operating system of each server; 4) Compare the indicator values ​​of each business module obtained in step 3) with the corresponding indicator thresholds of the business module. Based on the comparison results of each indicator value, obtain a comprehensive evaluation value. Determine the evaluation result of the domestically developed operating system based on the comprehensive evaluation value; wherein, according to Calculate the indicator values ​​for the business module, and then according to... Calculate the comprehensive evaluation value S; M of the business system. t Let Mt represent the metric value of the t-th business module, and M0 represent the initial evaluation value of the t-th business module. This indicates the weight of the adaptability evaluation index in the evaluation of the t-th business module, where n1 represents the number of adaptability evaluation indicators under the business module, and w i N represents the deduction weight for anomalies in the i-th suitability assessment metric. i This represents the number of times the i-th fitness assessment metric exceeds the threshold. This represents the weight of common software evaluation indicators in the evaluation of the t-th business module, where n2 represents the number of common software evaluation indicators under the business module, and w j N represents the deduction weight for the j-th software common evaluation index anomaly. j This indicates the number of times the j-th common software evaluation index exceeds the threshold and triggers an abnormal warning; This indicates the weight of the application business key evaluation indicators in the evaluation of the t-th business module, where n3 represents the number of application business key evaluation indicators under the business module, and w k N represents the deduction weight for anomalies in the key evaluation indicators of the k-th application business. k This represents the number of times the key evaluation indicator for the k-th application business exceeds the threshold and triggers an abnormal warning; m is the number of the business modules, and sw t Let be the weight coefficient of the t-th business module; The method for determining the threshold values ​​for each indicator in each business module is as follows: For the t-th business module, firstly, obtain the historical indicator data for each indicator in the t-th business module, and divide the historical indicator data for the indicator into historical indicator data with the attribute of working days and historical indicator data with the attribute of non-working days according to whether it is a working day; exclude abnormal data in the corresponding historical indicator data based on the historical indicator data of working days, and exclude abnormal data in the corresponding historical indicator data based on the historical indicator data of non-working days; then calculate the average value M of the historical data for the indicator based on the historical indicator data after excluding abnormal data; then set the threshold range of the indicator for the t-th business module to the interval (1-α)M~(1+α)M, where α is the set offset value.

2. The method according to claim 1, characterized in that, The compatibility evaluation indicators for the traffic processing service module when calculating the t-th service module include the version of the packet capture driver adapted to the domestic platform, the packet capture network card model, and the number of packet capture network ports. The common software evaluation indicators for the traffic processing service module when calculating the t-th service module include device manufacturer information, processor model, number of processor cores, total device memory size, device hard disk storage size, operating system version, CPU usage of the packet capture driver, CPU usage of the software, traffic volume of each network port, device load, hard disk read / write I / O, device hard disk utilization, device memory usage, packet capture driver running status and restart status, packet loss count of the packet capture driver, traffic received by the traffic processing software, packet loss count of the traffic processing software, memory consumption of the traffic processing software, single-core CPU usage of the traffic processing software, and service log content. The key application service evaluation indicators for the traffic processing service module when calculating the t-th service module include the packet capture driver adaptation version, packet capture network card model, number of packet capture network ports, packet loss count of the packet capture driver, traffic received by the traffic processing software, and packet loss count of the traffic processing software.

3. The method according to claim 1, characterized in that, The compatibility evaluation indicators for the data forwarding service module when calculating the t-th service module include the forwarding network card model, application software adaptation version, and the status of the resident port of the forwarding software. The common software evaluation indicators for the data forwarding service module when calculating the t-th service module include equipment manufacturer information, processor model, number of processor cores, total device memory size, device hard disk storage size, operating system version, operating system kernel version, device load, hard disk I / O read / write speed, device hard disk utilization, overall memory usage, CPU usage, forwarding network card bandwidth, forwarding network card packet loss, forwarding software CPU usage percentage, forwarding software throughput, and service operation response time. The key application service evaluation indicators for the data forwarding service module when calculating the t-th service module include the forwarding network card model, forwarding network card transmit / receive traffic, forwarding network card packet loss, the status of the resident port of the forwarding software, the forwarding software throughput, and service operation response time.

4. The method according to claim 1, characterized in that, The compatibility evaluation indicators for the data processing business module when calculating the t-th business module include the business data network card model, application software adaptation version, application software running status, and application software resident port status. The common software evaluation indicators for the data processing business module when calculating the t-th business module include equipment manufacturer information, processor model, number of processor cores, total device memory size, device hard disk storage size, operating system version, operating system kernel version, device load, hard disk I / O read / write speed, device hard disk utilization, overall memory usage, CPU usage, business operation log information, business data network card packet loss, business data network card bandwidth, application software memory consumption, application software CPU usage, and business response time. The key application business evaluation indicators for the data processing business module when calculating the t-th business module include business operation log information, business data network card packet loss, and business operation response time.

5. The method according to claim 1, characterized in that, The compatibility evaluation indicators for the data storage service module when calculating the t-th service module include the service data network card model, storage software running status, application software resident port status, storage software stress test results, and storage software performance benchmark test results. The common software evaluation indicators for the data storage service module when calculating the t-th service module include operational evaluation indicators such as device manufacturer information, processor model, number of processor cores, total device memory size, device hard disk storage size, operating system version, operating system kernel version, device load, hard disk I / O read / write speed, overall memory usage, CPU usage, service data network card bandwidth, service data network card traffic volume and packet loss, storage software memory consumption, storage software CPU usage, storage software data throughput, and storage software stress test and performance results. The key application service evaluation indicators for the data storage service module when calculating the t-th service module include the application software resident port status, storage software stress test results, and storage software performance benchmark test results.

6. The method according to claim 1, characterized in that, The adaptability evaluation indicators for the virtualization platform business module when calculating the t-th business module include the virtualization software version and the virtualization software running status; the common software evaluation indicators for the virtualization platform business module when calculating the t-th business module include equipment manufacturer information, processor model, number of processor cores, total device memory size, device hard disk storage size, operating system version, operating system kernel version, device load, hard disk I / O read / write speed, device hard disk utilization, overall memory usage, CPU usage, virtualization network bandwidth, virtualization platform software CPU usage and scheduling, software memory usage and scheduling, and software running status; the key application business evaluation indicators for the virtualization platform business module when calculating the t-th business module include virtualization network bandwidth and virtualization platform software CPU and memory usage and scheduling.

7. The method according to claim 1, characterized in that, The adaptability evaluation indicators for the cloud platform business module when calculating the t-th business module include cloud service architecture adaptability and cloud service architecture software running status; the common software evaluation indicators for the cloud platform business module when calculating the t-th business module include device manufacturer information, processor model, number of processor cores, total device memory size, device hard disk storage size, operating system version, operating system kernel version, device load, hard disk I / O read / write speed, device hard disk utilization, CPU usage, cloud platform system CPU usage, cloud platform system memory usage, cloud platform network bandwidth, and cloud platform data interaction volume; the key application business evaluation indicators for the cloud platform business module when calculating the t-th business module include cloud platform CPU and memory scheduling, data interaction volume, and platform network bandwidth.

Images