Software performance measurement method and system based on big data and grids

Abstract

The invention relates to a software performance measurement method and system based on big data and grids, and the method comprises the steps that each edge grid collects specified data of target software of at least one server device through a probe, stores the specified data, and treats the stored specified data, the specified data comprises index data, log data and link data; the software performance measurement center sends a data acquisition request to each edge grid, obtains the specified data from each edge grid, stores the specified data, and determines software performance according to the specified data, each edge grid is different from the network of the software performance measurement center, and the software performance measurement center is based on a multi-cloud network technology. Software data of different grids and software performance data of different edges can be collected, accurate, unified and efficient data collection is achieved, and therefore efficient measurement of software performance of multiple pieces of software in different grids is achieved.

Description

Technical Field

[0001] The present disclosure relates to the field of big data technologies, and in particular, to a software performance measurement method and system based on big data and grids. Background Art

[0002] Currently, in the context of mainstream new technologies such as big data, artificial intelligence, and cloud computing, during the continuous progress of production operations, governments and enterprises always need to ensure the efficient management and stable operation of application software, accurately locate fault information, ensure the real-time security of device networks, and prevent network attacks, etc. Software performance measurement solutions have emerged as the times require, and this method will have a profound impact on various fields.

[0003] With the rapid expansion of the businesses of governments and enterprises, in order to ensure the stable and normal operation of each business, different cloud networks and isolation strategies are usually established to make each business not interfere with each other, which prolongs the time of stable operation of the business. On the contrary, such network isolation brings many disadvantages to operation and maintenance. The intelligent operation and maintenance system cannot automatically or quickly obtain the software performance data at the edge. At the same time, each edge network structure also has levels and cannot handle various complex scenarios, bringing many difficulties to the work of operation and maintenance. In addition, with the gradual increase of operation and maintenance software, after a large amount of software performance data is collected, there are too many dirty data or useless data, bringing huge challenges to the efficient monitoring of software performance. Summary of the Invention

[0004] In order to solve the above technical problems or at least partially solve the above technical problems, embodiments of the present disclosure provide a software performance measurement method and system based on big data and grids.

[0005] In a first aspect, embodiments of the present disclosure provide a software performance measurement method based on big data and grids, which is characterized in that it is applied to a software performance measurement system. The software performance measurement system includes a software performance measurement center and a plurality of edge grids. The method includes:

[0006] Each edge grid uses a probe to collect specified data of a target software of at least one server device, stores the specified data, and performs governance on the stored specified data, where the specified data includes metric data, log data, and link data;

[0007] The software performance measurement center sends a data collection request to each edge grid, obtains the specified data from each edge grid, stores the specified data, and determines software performance according to the specified data, where each edge grid has a different network from the software performance measurement center.

[0008] In a possible implementation manner, determining the software performance according to the specified data includes:

[0009] Convert the log data into standard log data;

[0010] Perform data processing on the specified data to eliminate null and invalid data;

[0011] According to the timestamp, perform metric correlation on the filtered metric data, standard log data, and link data for three-dimensional measurement after an alarm occurs;

[0012] Set up a monitoring model to define the specified ranges of the metric data, standard log data, and link data to be measured. Among them, the monitoring model is used to monitor whether any of the metric data, standard log data, and link data exceeds the specified range, and an alarm is issued when any of the metric data, standard log data, and link data exceeds the specified range;

[0013] In the case of an alarm, if all the associated metric data, standard log data, and link data are abnormal, determine that the alarm is a high risk;

[0014] In the case of no alarm, according to the scheduled task, perform real-time comparative analysis on the metric data, standard log data, and link data within a preset time period, and issue a performance monitoring report for performance optimization.

[0015] In a possible implementation manner, the software performance measurement center includes a grid server, and the edge grid includes a grid client. The method further includes:

[0016] Before the software performance measurement center sends a data collection request to each edge grid, establish a connection pipeline between the grid client and the grid server,

[0017] The software performance measurement center sending a data collection request to each edge grid includes:

[0018] The grid server listens in real time for the data collection request of the software performance measurement center for the edge grid, and sends the listened data collection request to the grid client.

[0019] In a possible implementation manner, the establishing a connection pipeline between the grid client and the grid server includes:

[0020] Provide a client connection component deployment package for the grid client;

[0021] After deploying the client connection component in the grid client, the grid server receives a connection request from the client connection component, verifies the connection request, and sends a verification success message to the grid client;

[0022] The grid server receives a channel establishment request initiated by the grid client in response to the verification success information, maps out a virtual proxy address corresponding to the grid client, and realizes communication between the grid server and the grid client through data interaction between the virtual proxy address and the client connection component.

[0023] In a possible implementation manner, the edge grid includes a data reporting service. The software performance measurement center sends data collection requests to each edge grid and obtains the specified data from each edge grid, including:

[0024] The grid server sends the data collection request to the grid client corresponding to each edge grid;

[0025] The grid client receives the data collection request, invokes the data reporting service, extracts the data corresponding to the data collection request, and transmits it to the grid server.

[0026] In a possible implementation manner, the governance of the stored specified data includes:

[0027] Filter the stored specified data according to preset keywords, and remove duplicates of null data from the filtered data;

[0028] Sort the data after duplicate removal according to the reporting priority.

[0029] In a possible implementation manner, the storage of the specified data includes:

[0030] Store the specified data in full volume in different types of databases according to different formats;

[0031] Monitor and collect the storage status of each database in real time, collect and obtain migration bottleneck status data information, and detect cache adjustment trend data information in real time;

[0032] Clean and extract features from the migration bottleneck status data information and the cache adjustment trend data information, and standardize the processed data information according to the dimensionless processing technology;

[0033] Comprehensively analyze and learn the migration bottleneck status data information after data cleaning to obtain bottleneck factors, and combine the number of interruptions that occur during data migration to summarize and generate a migration status coefficient;

[0034] Through the cache adjustment trend data information after feature extraction, obtain the block size and cache hit rate, associate the block size with the cache hit rate, obtain a cache regulation coefficient, comprehensively analyze the cache regulation coefficient and the migration status coefficient, and combine with a neural network model for post-training to generate a software storage performance prediction model, and after dimensionless processing, obtain a performance trend evaluation index;

[0035] Compare and analyze the preset evaluation threshold with the performance situation evaluation index to obtain a prediction result, and obtain a corresponding level improvement strategy for the prediction result.

[0036] In a possible implementation manner, the performance situation evaluation index is obtained through the following formula:

[0037]

[0038] Wherein, TSzs represents the performance situation evaluation index, Ttlz represents the throughput, F1, F2, and F3 respectively represent the proportionality coefficients of the cache regulation coefficient Tkxs, the migration status coefficient Pyxs, and the throughput Ttlz. Among them, 0.10 ≤ F1 ≤ 0.22, 0.15 ≤ F2 ≤ 0.36, 0.20 ≤ F3 ≤ 0.42, and 0.55 ≤ F1 + F2 + F3 ≤ 1.0, and C represents the first correction constant.

[0039] In a possible implementation manner, the migration status coefficient is obtained through the following formula:

[0040]

[0041] Wherein, Pyxs represents the migration status coefficient, Ysjz represents the amount of migrated data, both α and β represent proportionality coefficients, and M1 represents the third correction constant.

[0042] In a second aspect, an embodiment of the present disclosure provides a software performance measurement system, including a software performance measurement center and a plurality of edge grids. Among them,

[0043] Each edge grid uses a probe to collect specified data of the target software of at least one server device, stores the specified data, and governs the stored specified data. Among them, the specified data includes metric data, log data, and link data;

[0044] The software performance measurement center sends a data collection request to each edge grid, obtains the specified data from each edge grid, stores the specified data, and determines the software performance according to the specified data. Among them, the network of each edge grid is different from that of the software performance measurement center.

[0045] The above technical solutions provided by the embodiments of the present disclosure have at least some or all of the following advantages compared with the prior art:

[0046] In the software performance measurement method based on big data and grid according to the embodiments of the present disclosure, each edge grid uses a probe to collect specified data of the target software of at least one server device, stores the specified data, and governs the stored specified data, where the specified data includes metric data, log data, and link data; the software performance measurement center sends a data collection request to each edge grid, obtains the specified data from each edge grid, stores the specified data, and determines software performance according to the specified data. Among them, the network of each edge grid is different from that of the software performance measurement center. Based on multi-cloud network technology, it can collect software data of different grids and software performance data of different edges, realizing accurate, unified, and efficient data collection, so as to achieve efficient measurement of the software performance of multiple softwares in different grids. Description of the Drawings

[0047] The drawings here are incorporated into the specification and constitute a part of this specification, showing the embodiments in line with the present disclosure, and are used together with the specification to explain the principles of the present disclosure.

[0048] In order to more clearly illustrate the technical solutions in the embodiments of the present disclosure or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or related technologies. Obviously, for those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0049] Figure 1 Schematically shows a schematic flowchart of the software performance measurement method based on big data and grid according to the embodiments of the present disclosure;

[0050] Figure 2 Schematically shows a schematic flowchart of data collection between the software performance measurement center and multiple edge grids according to the embodiments of the present disclosure;

[0051] Figure 3 Schematically shows a schematic flowchart of the three-dimensional software performance measurement method of the software performance measurement center according to the embodiments of the present disclosure;

[0052] Figure 4 Schematically shows a schematic interaction flowchart between application software A in the edge grid and the software performance measurement center according to the embodiments of the present disclosure;

[0053] Figure 5 Schematically shows a schematic structural diagram of the software performance measurement system according to the embodiments of the present disclosure. Detailed Embodiments

[0054] To make the objectives, technical solutions, and advantages of the embodiments of the present disclosure clearer, the following will clearly and completely describe the technical solutions in the embodiments of the present disclosure with reference to the accompanying drawings in the embodiments of the present disclosure. Obviously, the described embodiments are some, but not all, of the embodiments of the present disclosure. All other embodiments obtained by those of ordinary skill in the art based on the embodiments in the present disclosure without creative efforts fall within the scope of protection of the present disclosure.

[0055] Referring to Figure 1 , an embodiment of the present disclosure provides a software performance measurement method based on big data and grids, which is applied to a software performance measurement system. The software performance measurement system includes a software performance measurement center and multiple edge grids. The method includes:

[0056] S1. Each edge grid uses a probe to collect specified data of the target software of at least one server device, stores the specified data, and performs governance on the stored specified data, where the specified data includes metric data, log data, and link data.

[0057] In some embodiments, an edge grid refers to connecting various resources into a logical whole using the Internet to provide users with integrated information and application services. Users can share resources and transfer data in this area; multiple edge networks are multiple network areas with isolated resources, and each network area can communicate through only one channel to reduce the security risks of each area.

[0058] S2. The software performance measurement center sends a data collection request to each edge grid, obtains the specified data from each edge grid, stores the specified data, and determines the software performance based on the specified data, where each edge grid has a different network from the software performance measurement center.

[0059] In this embodiment, in step S2, the determining the software performance based on the specified data includes:

[0060] Converting the log data into standard log data;

[0061] Performing data processing on the specified data to eliminate empty and invalid data;

[0062] Performing metric association on the filtered metric data, standard log data, and link data according to the timestamp for three-dimensional measurement after an alarm occurs;

[0063] Set up a monitoring model, define the specified ranges of the metric data, standard log data, and link data to be measured. Among them, the monitoring model is used to monitor whether any one of the metric data, standard log data, and link data exceeds the specified range. In the case where any one of the metric data, standard log data, and link data exceeds the specified range, an alarm is issued;

[0064] In the case of an alarm, if all the associated metric data, standard log data, and link data are abnormal, determine that the alarm is a high risk;

[0065] In the case where no alarm occurs, according to the scheduled task, the metric data, standard log data, and link data within a preset time period are compared and analyzed in real time, and a performance monitoring report is issued for performance optimization.

[0066] In this embodiment, the software performance measurement center includes a grid server, and the edge grid includes a grid client. Before step S2, the method further includes:

[0067] Establish a connection pipeline between the grid client and the grid server,

[0068] In step S2, the software performance measurement center sends a data collection request to each edge grid, including:

[0069] The grid server listens in real time for the data collection request of the software performance measurement center for the edge grid, and sends the listened data collection request to the grid client.

[0070] In this embodiment, establishing a connection pipeline between the grid client and the grid server includes:

[0071] Provide a client connection component deployment package for the grid client;

[0072] After the client connection component is deployed in the grid client, the grid server receives a connection request from the client connection component, verifies the connection request, and sends a verification success message to the grid client;

[0073] The grid server receives the channel establishment request initiated by the grid client in response to the verification success message, maps out a virtual proxy address corresponding to the grid client, and realizes the communication between the grid server and the grid client through the data interaction with the client connection component using the virtual proxy address.

[0074] See Figure 2, The data collection of multiple edge grids can collect the software performance data of multiple private clouds or intranets, and report the data to the software performance measurement center in the form of pipeline services. Each grid can build its own data reporting system and report layer by layer to the software performance measurement center, so as to realize orderly and highly decoupled data collection. The specific process is as follows:

[0075] The data collection of multiple edge grids depends on the grid clients of each edge grid and the grid server of the software performance measurement center. Among them, the grid clients can be deployed on the proxy servers in the edge grids. After the grid clients are started, they actively connect to the grid server and establish a long connection pipeline to be responsible for communication between networks. When the channel is disconnected due to abnormal reasons, it will retry the connection; after the long connection pipeline is established, the grid server will create a listening service, which is used to obtain the data collection requests of the software performance measurement center for the software performance data of multiple edge grids in real time; when the grid server monitors the data collection request, the grid server sends the request to the grid client, and after the grid client receives the request, it reports the data to the grid server, that is, the software performance measurement center, to realize the aggregation of software performance data.

[0076] The software performance measurement method based on big data and grids in the present disclosure, based on multi-cloud network technology and self-developed pipeline technology, can collect software data of different edge grids, software performance data of different edge grids, build multi-level data reporting and collection channels, and achieve accurate, unified and efficient data collection processes. It can solve the problem in the prior art that in a complex environment, after separately developing the specified data to form an interface and docking the interface to the software performance measurement center end, it is impossible to report data layer by layer in an automated form during the data collection stage, there are faults between the networks for data reporting, and a large amount of customized development work is required. Among them, the complex environment refers to the situation where the networks between the software performance measurement center and each edge grid are different, and the grids between different edge grids are also different, realizing an architecture system for software operation and maintenance across networks, regions and spaces, and bringing scalability and basic support to software performance monitoring and operation and maintenance.

[0077] In this embodiment, in step S1, the governance of the specified data stored includes:

[0078] Filter the specified data stored according to the preset keywords, and remove duplicates of the filtered data;

[0079] Sort the data after duplicate removal according to the reporting priority.

[0080] In this embodiment, in step S2, the edge grid includes a data reporting service, and the software performance measurement center sends a data collection request to each edge grid and obtains the specified data from each edge grid, including:

[0081] The grid server sends the data collection request to the grid client corresponding to each edge grid;

[0082] The grid client receives the data collection request, calls the data reporting service, extracts the data corresponding to the data collection request, and transmits it to the grid server.

[0083] See Figure 2 , data aggregation, governance, filtering, and priority reporting settings can be performed on the proxy servers of each edge grid. When the server structure and data collection volume of an edge grid are particularly large, it is necessary to manage the dirty data in the edge grid in advance. In addition to deploying grid servers for communication with the software performance measurement center, the proxy servers of each edge grid can also serve as the data center of the edge grid, query data through a visualization page, and set filtering policies and reporting priority policies for the data, preventing the data reporting channel from being blocked due to a sharp increase in the data volume, and unable to detect problems and alarms in a timely manner, which brings problems to the normal operation of the business. It avoids the problems in the prior art that the collected data cannot be partially aggregated, and after being managed by big data technology and reported to the intelligent software performance measurement center, a large amount of garbage data may be stored, which brings great obstacles to the maintenance personnel to handle problems, and the problems that due to the inability to set synchronization priorities and synchronization time intervals according to the importance of the data, effective differential monitoring cannot be formed, and key monitoring data cannot be obtained in a timely manner due to a sharp increase in the data volume.

[0084] The software performance measurement method based on big data and grid of the present disclosure is based on big data technology, integrates multiple links such as data governance, data filtering, and data reporting, and performs hierarchical reporting and governance on software performance data, achieving differential, reasonable, and effective monitoring of data.

[0085] In this embodiment, in steps S1 and S2, the specified data is stored in different types of databases in full volume according to different formats;

[0086] Perform real-time monitoring and collection on the storage status of each database, collect and obtain migration bottleneck status data information, and detect cache adjustment trend data information in real time;

[0087] Perform data cleaning and feature extraction on the migration bottleneck status data information and the cache adjustment trend data information, and perform standardization processing on the processed data information according to the dimensionless processing technology;

[0088] Comprehensively analyze and learn the data information of the migration bottleneck state after data cleaning to obtain bottleneck factors, and combine the number of interruptions that occur during data migration to summarize and generate a migration state coefficient;

[0089] Through the cache adjustment situation data information after feature extraction, obtain the block size and cache hit rate, associate the block size with the cache hit rate, obtain a cache regulation coefficient, comprehensively analyze the cache regulation coefficient and the migration state coefficient, and combine with a neural network model to train and generate a software storage performance prediction model, and after dimensionless processing, obtain a performance situation evaluation index;

[0090] Compare and analyze a preset evaluation threshold with the performance situation evaluation index to obtain a prediction result, and obtain a corresponding level improvement strategy for the prediction result.

[0091] In this embodiment, the performance situation evaluation index is obtained through the following formula:

[0092]

[0093] Among them, TSzs represents the performance situation evaluation index, Ttlz represents the throughput, F1, F2, and F3 respectively represent the proportionality coefficients of the cache regulation coefficient Tkxs, the migration state coefficient Pyxs, and the throughput Ttlz. Among them, 0.10 ≤ F1 ≤ 0.22, 0.15 ≤ F2 ≤ 0.36, 0.20 ≤ F3 ≤ 0.42, and 0.55 ≤ F1 + F2 + F3 ≤ 1.0, and C represents the first correction constant.

[0094] In this embodiment, the migration state coefficient is obtained through the following formula:

[0095]

[0096] Among them, Pyxs represents the migration state coefficient, Ysjz represents the amount of migrated data, α and β both represent proportionality coefficients, and M1 represents the third correction constant.

[0097] In this embodiment, the cache regulation coefficient is obtained through the following formula:

[0098]

[0099] Among them, Tkxs represents the cache regulation coefficient, RLz represents the cache capacity, Kds represents the block size, Hcmz represents the cache hit rate, a1 and a2 both represent proportionality coefficients, and M2 represents the fourth correction constant.

[0100] See Figure 2, the specified data of each edge grid is uniformly reported to the proxy server of the edge grid through data aggregation. Then, the proxy stores all the data, and through data governance and filtering, the valid data is aggregated to the software performance measurement center. The software performance measurement center stores this data and performs hot and cold storage based on the validity and timeliness of the data. Based on the two-way storage of big data, it can not only provide hot and cold storage modes, but also provide the data storage mode between the software performance measurement center and the edge, realizing double guarantee of data.

[0101] In the software performance measurement method based on big data and grid of the present disclosure, after the software performance data is reported, the data is stored at the software performance measurement center, while the edge grid side performs backup and cold and heat treatment on the data, which can avoid the risk of data loss caused by the specified data being stored only at the software performance measurement center in the prior art, as well as the problems that the centralized storage of data easily occupies a large amount of storage resources and the specified data of different edge grids cannot be accessed in modules.

[0102] See Figure 3 , a data aggregation analysis and measurement method for indicators, logs, and links in three dimensions is provided. First, the three types of software performance data are aggregated and classified. Second, the data in each dimension is analyzed and visually displayed. Then, a single alarm or combined alarm strategy is set. Not only is an alarm issued through simple indicator data, but by combining log and link data, all problem root causes can be accurately located, improving the operation and maintenance efficiency. It can solve the problems in the prior art that an alarm strategy is set for the collected specified data and then an alarm is issued, only paying attention to the abnormal conditions of the existing software, the data types concerned are relatively single, not all specified data types can be completely covered, only single indicator, log, or link data is provided, and the aggregation or discrete analysis of data for a period of time cannot achieve full-stack and full-angle periodic monitoring, nor can the performance and operation conditions of the software be comprehensively analyzed and measured.

[0103] The software performance measurement method based on big data and grid of the present disclosure provides full-stack three-dimensional integrated software performance measurement and situation analysis based on the measurement of three-dimensional monitoring data, aggregates and discretely analyzes the software monitoring data, effectively operates and maintains the entire life cycle of the software, quickly locates problems, discovers performance defects, and timely adjusts software performance parameters.

[0104] See Figure 4 , the interaction process between application software A in the edge grid and the software performance measurement center is as follows:

[0105] The first step is edge grid data collection:

[0106] First, deploy probes on the monitored devices in the edge grid to collect metrics, logs, and link data of software or applications on the servers in the edge grid. Taking application A as an example, collect the log data of application A on the current day, and deploy the corresponding probes to the corresponding log directories to obtain the data;

[0107] Converge the collected data to the proxy server side of the edge grid, and push the collected log data of application A on the current day to the proxy server through the data reporting service;

[0108] The proxy server stores all the collected data in different types of databases in different formats;

[0109] After the data is stored in the database, data governance, filtering, and reporting rules can be set. Taking the log of application A on the current day as an example, after the data is stored in the database, keyword filtering, duplicate removal of null data, and setting the data reporting priority can be performed on the log data.

[0110] Second step, data collection method based on multiple grids:

[0111] This data collection is not only within the network of the edge grid, where software performance data of monitored devices and software with different network isolations can be collected, but also the data collection and reporting of each grid can be customized, thus achieving the effect of cross-network, cross-device, and cross-region data collection:

[0112] The software performance measurement center sends a request to collect the log of application A on the current day, and forwards the request message to the grid server;

[0113] The grid server and the grid client always keep the channel connected, and the grid server sends the requested message to the grid client corresponding to application A through this technology;

[0114] After receiving the requirement to collect the log of application A on the current day, the grid client calls the data reporting service, extracts the corresponding log data according to different passed parameters and data governance, filtering and other strategies, and then transmits it to the software performance measurement center through the channel.

[0115] Third step, data storage and software performance measurement:

[0116] After receiving the log data of application A, the software performance measurement center stores the data in the hot database for timely call, and then transfers it to the cold database according to the hot and cold storage strategy;

[0117] A large amount of and continuous metrics, logs, and link data of application A will be stored in the database, and the data will be analyzed and displayed through a three-dimensional performance measurement method;

[0118] Through the three-dimensional performance measurement method, compare the analyzed data with the set alarm policies to generate software faults or performance alarms, and provide corresponding decision-making data to help locate problems, or bind through alarms, set automated handling commands and scripts for automated fault handling; faults and alarms can also be assigned to relevant personnel for offline processing in the form of work orders and notifications.

[0119] In this embodiment, determining the performance of application software A according to the specified data includes:

[0120] Regularly extract the latest specified data of application A from the data source corresponding to the specified data, where the specified data includes metric data, log data, and link data;

[0121] Convert the log data into standard log data;

[0122] Set up a monitoring model to define the ranges of metric data, standard log data, and link data to be measured;

[0123] Process the specified data to eliminate empty and invalid data;

[0124] Perform metric association on the metric data, standard log data, and link data for three-dimensional measurement after an alarm occurs;

[0125] Visually display the real-time three-dimensional monitoring data, and set the alarm policy. In the case of an alarm, calculate according to the associated three-dimensional data. If all the associated data is abnormal, judge the alarm as a serious risk. In the case of no alarm, the three-dimensional data calculation model performs real-time comparative analysis on the monitoring performance data within a certain time period according to the scheduled task, and issues a performance monitoring report for performance optimization.

[0126] The disclosed software performance measurement method based on big data and grids, through communication channels, overcomes the existing problems of network isolation, which make data collection difficult and operation and maintenance monitoring unscalable. This has resulted in the inability to converge software operation and maintenance across multiple grids onto a single platform, or the need for extensive manual adjustments to provide monitoring data. This method significantly improves the convenience of operation and maintenance data collection. Big data governance addresses the issues of disordered and full data reporting. Data is effectively reported at the edge after data processing, preventing monitoring delays caused by data surges or network jitter that impact the normal operation of business systems. Three-dimensional software performance measurement technology addresses issues such as insufficient software performance parameters, ineffective software monitoring, and long traceability of abnormal issues. Software health and root causes are comprehensively analyzed from three perspectives, addressing the issue of alarm jitter caused by insufficient data and incomplete analysis. It also provides a visual display of issues that might otherwise remain undetected due to the depth of the root cause. Tiered data storage ensures data security, prevents software operation and maintenance stagnation caused by data loss at the center, and improves data utilization by performing hot and cold data processing.

[0127] The software performance measurement method based on big data and grid disclosed in the present invention can provide unified data collection, storage, governance, monitoring, alarm, disposal, analysis and display, etc. with the help of big data technology and multi-grid intercommunication technology, breaking the original operation and maintenance islands, establishing standard operation and maintenance specifications and systems, completing software performance data integration and capability improvement, relying on multi-scenario operation and maintenance adaptation with intelligent operation and maintenance as the core, maintaining stable and efficient operation of business and operation and maintenance work, and providing a bottom-up standard governance process through hierarchical management of monitoring data. The data-driven operation and maintenance system architecture improves the security of equipment monitoring data transmission, reduces the occupation of network bandwidth resources, reduces the cost of use, and opens the way for digitalization and digital intelligence transformation of operation and maintenance.

[0128] See also Figure 5 , an embodiment of the present disclosure provides a software performance measurement system, comprising a software performance measurement center and a plurality of edge grids, wherein,

[0129] Each edge grid uses a probe to collect specified data of target software of at least one server device, stores the specified data, and manages the stored specified data, wherein the specified data includes indicator data, log data, and link data;

[0130] The software performance measurement center sends a data collection request to each edge grid, obtains the specified data from each edge grid, stores the specified data, and determines software performance based on the specified data, wherein each edge grid is a different network from the software performance measurement center.

[0131] For the implementation processes of the functions and roles of each unit in the above system, please refer to the implementation processes of the corresponding steps in the above method for details, which will not be elaborated here.

[0132] For the system embodiment, since it basically corresponds to the method embodiment, the relevant parts can be referred to the partial description of the method embodiment. The system embodiment described above is only illustrative. The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed to multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of the present invention. Those of ordinary skill in the art can understand and implement it without creative efforts.

[0133] It should be noted that in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the term "comprising", "including" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or device comprising a series of elements not only includes those elements, but also includes other elements not expressly listed, or elements inherent to such process, method, article or device. Without further limitation, the element defined by the statement "comprising an..." does not exclude the existence of additional identical elements in the process, method, article or device comprising the element.

[0134] The above are only specific implementation manners of the present disclosure, enabling those skilled in the art to understand or implement the present disclosure. Various modifications to these embodiments will be obvious to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present disclosure. Therefore, the present disclosure will not be limited to these embodiments shown herein, but will be accorded the widest scope consistent with the principles and novel features claimed herein.

Claims

1. A software performance measurement method based on big data and grid, characterized in that, Applied to a software performance measurement system, the software performance measurement system includes a software performance measurement center and multiple edge grids, and the method includes: Each edge grid uses a probe to collect specified data of the target software of at least one server device, stores the specified data, and governs the stored specified data, where the specified data includes metric data, log data, and link data; The software performance measurement center sends a data collection request to each edge grid, obtains the specified data from each edge grid, stores the specified data, and determines the software performance according to the specified data, where the network of each edge grid is different from that of the software performance measurement center.

2. The method according to claim 1, characterized in that, The determining the software performance according to the specified data includes: Converting the log data into standard log data; Performing data processing on the specified data to eliminate empty and invalid data; Performing metric association on the filtered metric data, standard log data, and link data according to the timestamp for three-dimensional measurement after an alarm occurs; Setting up a monitoring model to define the specified ranges of the metric data, standard log data, and link data to be measured, where the monitoring model is used to monitor whether any of the metric data, standard log data, and link data exceeds the specified range, and an alarm is issued when any of the metric data, standard log data, and link data exceeds the specified range; In the case of an alarm, if all the associated metric data, standard log data, and link data are abnormal, determine that the alarm is a high risk; In the case of no alarm, according to the scheduled task, compare and analyze the metric data, standard log data, and link data in a preset time period in real time, and issue a performance monitoring report for performance optimization.

3. The method according to claim 1, wherein The software performance measurement center includes a grid server, and the edge grid includes a grid client, and the method further includes: Before the software performance measurement center sends a data collection request to each edge grid, establish a connection pipeline between the grid client and the grid server, The software performance measurement center sending a data collection request to each edge grid includes: The grid server listens for the data collection request of the software performance measurement center for the edge grid in real time and sends the listened data collection request to the grid client.

4. The method according to claim 3, characterized in that, The establishing a connection pipeline between the grid client and the grid server includes: Providing a client connection component deployment package for the grid client; After deploying the client connection component in the grid client, the grid server receives a connection request from the client connection component, verifies the connection request, and sends a verification success message to the grid client; The grid server receives the channel establishment request initiated by the grid client in response to the verification success message, maps out a virtual proxy address corresponding to the grid client, and realizes the communication between the grid server and the grid client through the data interaction with the client connection component at the virtual proxy address.

5. The method according to claim 3, wherein The edge grid includes a data reporting service, and the software performance measurement center sending a data collection request to each edge grid and obtaining the specified data from each edge grid includes: The grid server sends data collection requests to the grid clients corresponding to each edge grid; The grid client receives the data collection request, invokes the data reporting service, extracts the data corresponding to the data collection request, and transmits it to the grid server.

6. The method according to claim 1, wherein The governance of the specified stored data includes: Filtering the specified stored data according to a preset keyword, and removing duplicates of null data from the filtered data; Sorting the reported priorities of the data after duplicate removal.

7. The method according to claim 1, characterized in that, The storing of the specified data includes: Storing the specified data in full volume in different types of databases in different formats; Performing real-time monitoring and collection of the storage conditions of each database, collecting and obtaining migration bottleneck status data information, and detecting cache adjustment trend data information in real time; Performing data cleaning and feature extraction on the migration bottleneck status data information and the cache adjustment trend data information, and performing standardization processing on the processed data information according to the dimensionless processing technology; Performing comprehensive analysis and learning on the migration bottleneck status data information after data cleaning to obtain bottleneck factors, and combining the number of interruptions occurring during data migration to summarize and generate a migration status coefficient; Obtaining the block size and cache hit rate through the cache adjustment trend data information after feature extraction, associating the block size with the cache hit rate to obtain a cache regulation coefficient, performing comprehensive analysis on the cache regulation coefficient and the migration status coefficient, and combining with a neural network model to train and generate a software storage performance prediction model, and after dimensionless processing, to obtain a performance trend evaluation index; Comparing and analyzing a preset evaluation threshold with the performance trend evaluation index to obtain a prediction result, and obtaining a corresponding level improvement strategy for the prediction result.

8. The method according to claim 7, characterized in that The performance trend evaluation index is obtained through the following formula: where, TSzs represents the performance trend evaluation index, Ttlz represents the throughput, F1, F2, and F3 respectively represent the proportionality coefficients of the cache regulation coefficient Tkxs, the migration status coefficient Pyxs, and the throughput Ttlz, where 0.10 ≤ F1 ≤ 0.22, 0.15 ≤ F2 ≤ 0.36, 0.20 ≤ F3 ≤ 0.42, and 0.55 ≤ F1 + F2 + F3 ≤ 1.0, and C represents the first correction constant.

9. The method according to claim 7, wherein The migration status coefficient is obtained through the following formula: where, Pyxs represents the migration status coefficient, Ysjz represents the amount of migrated data, α and β both represent proportionality coefficients, and M1 represents the third correction constant.

10. A software performance measurement system, characterized in that, It includes a software performance measurement center and multiple edge grids, where Each edge grid uses a probe to collect specified data of the target software of at least one server device, stores the specified data, and governs the stored specified data, where the specified data includes metric data, log data, and link data; The software performance measurement center sends data collection requests to each edge grid, obtains the specified data from each edge grid, stores the specified data, and determines software performance according to the specified data, where the network of each edge grid is different from that of the software performance measurement center.

Images