Big data system configuration parameter monitoring and tuning system based on deep learning

Abstract

The present invention relates to the technical field of big data system configuration optimization, and is used to solve the problem that the existing monitoring and tuning methods for big data system configuration parameters have great lag and inaccuracy, cannot achieve real-time monitoring and analysis of the big data system configuration, and therefore are difficult to truly achieve accurate optimization of the big data system configuration parameters, which hinders the stability and efficient development of the big data system. In particular, a big data system configuration parameter monitoring and tuning system based on deep learning is disclosed, comprising a monitoring optimization and analysis platform, wherein a server is arranged inside the monitoring optimization and analysis platform, and the server is communicatively connected with a data acquisition unit, a monitoring and analysis unit, a grade quantification unit, an optimization control unit, a tuning evaluation unit and a display terminal; the present invention, while realizing real-time monitoring and analysis of the configuration parameters of the big data system, also realizes accurate optimization of the configuration parameters of the big data system, and promotes the efficient development of the big data system.

Description

Technical Field

[0001] The present invention relates to the technical field of big data system configuration optimization, and in particular to a big data system configuration parameter monitoring and tuning system based on deep learning. Background Art

[0002] Big data is a product of the high-tech era. Big data analysis can help companies make smarter decisions. For example, through big data analysis, companies can conduct precision marketing, timely analyze the root causes of faults, problems and defects, or use clickstream analysis and data mining to avoid fraud, etc. Therefore, more and more companies will actively embrace big data, actively understand and use big data technology, lay the foundation for future market development, and improve the company's business in an effective way. Therefore, whether from the perspective of corporate development or social development, the application of big data systems has become increasingly important;

[0003] Big data systems are used to integrate and analyze massive amounts of data information. Therefore, the requirements for monitoring and tuning the configuration parameters of big data systems are particularly high. In the existing monitoring and tuning of the configuration parameters of big data systems, most of them are performed when the big data system is running stuck. The technicians use auxiliary monitoring software to perform compensatory monitoring and regulation of the configuration of the big data system. The monitoring and regulation method has great lag and inaccuracy, and it is impossible to achieve real-time monitoring and analysis of the configuration of the big data system. Therefore, it is difficult to truly achieve accurate optimization of the configuration parameters of the big data system, which hinders the stable and efficient development of the big data system.

[0004] In order to solve the above defects, a technical solution is now provided. Summary of the invention

[0005] The purpose of the present invention is to solve the problem that the existing monitoring and tuning methods for the configuration parameters of the big data system have great lag and inaccuracy, and it is impossible to achieve real-time monitoring and analysis of the configuration of the big data system. Therefore, it is difficult to truly realize the accurate optimization of the configuration parameters of the big data system, which hinders the stable and efficient development of the big data system. By using the symbol calibration, formulaic processing, establishment of a dynamic coordinate system and signal comparison output, the configuration parameters of the big data system at different levels are monitored and analyzed in real time. By data summation comparison, signal output and classification instruction output, accurate optimization analysis of the configuration parameters of the big data system is realized, and accurate tuning processing of the big data system configuration is performed by executing the optimization instructions item by item. By using the symbolic calibration, iterative call of data, formulaic analysis and numerical comparison output, the configuration parameters of the big data system at the unit time level after optimization are optimized, iterative training and analysis processing are performed, thereby improving the real-time monitoring and analysis of the configuration parameters of the big data system, and truly realizing the accurate optimization of the configuration parameters of the big data system, promoting the stable and efficient development of the big data system, and proposing a big data system configuration parameter monitoring and tuning system based on deep learning.

[0006] The purpose of the present invention can be achieved through the following technical solutions:

[0007] A big data system configuration parameter monitoring and tuning system based on deep learning includes a monitoring optimization and analysis platform, wherein a server is provided inside the monitoring optimization and analysis platform, and the server is connected to a data acquisition unit, a monitoring and analysis unit, a grade quantitative unit, an optimization control unit, a tuning evaluation unit and a display terminal;

[0008] The monitoring, optimization and analysis platform is used to monitor and optimize the configuration parameter information of the big data system. The configuration characteristic parameter information of the big data system is obtained in real time through the data acquisition unit, and sent to the monitoring and analysis unit. The monitoring and analysis unit is used to receive the configuration characteristic parameter information of the big data system, and perform state monitoring and analysis processing based on it, and generate forward matching signals, reverse matching signals and non-matching signals based on it, and send them all to the level quantitative unit. The level quantitative unit performs optimization and fixed-gear analysis processing on each type of matching signal received, and generates each type of extended gear position instruction based on it, and sends them all to the optimization and control unit.

[0009] The optimization and control unit is used to receive various types of extended gear instructions, and perform configuration optimization analysis and processing on the configuration of the big data system based on this, generate an optimization completion signal based on this, and send it to the tuning and evaluation unit. The tuning and evaluation unit is used to receive the optimization completion signal, and retrieve the configuration feature parameter information of the big data system at the unit time level after optimization based on this, and perform optimization iterative training analysis and processing based on this, generate effective optimization signals and invalid optimization signals based on this, and send them to the display terminal for display in the form of text description.

[0010] Furthermore, the specific operation steps of condition monitoring analysis and processing are as follows:

[0011] S1: Obtain the process value and memory value in the configuration characteristic parameter information of the unit time level big data system in real time, and mark them as che i and nec i , where i is a positive integer greater than or equal to 1, the process value and memory value are normalized to obtain the running performance coefficient yux at each time point i ;

[0012] S2: According to step S1, randomly capture the operating performance coefficient yux of any period of time in the unit time level i The performance dynamic coordinate system is established based on the time as the horizontal axis and the operating performance coefficient as the vertical axis. The operating performance coefficient yux for a period of time is plotted by plotting points. i Outlined on a performance dynamic coordinate system, and based on this, a type of model analysis processing is performed, and a superior operating performance signal and a inferior operating performance signal are generated;

[0013] S3: Real-time capture of the read and write values ​​in the configuration characteristic parameter information of the big data system in the same time period as step S2, and calibrate it as dux i , with time as the horizontal axis and the reading and writing value as the vertical axis, a dynamic reading and writing coordinate system is established based on this, and the reading and writing value dux over a period of time is plotted by drawing points. i Outlined on the read-write dynamic coordinate system, and based on this, perform two-type model analysis and processing, and generate a signal of superior read-write performance and a signal of inferior read-write performance;

[0014] S4: According to steps S2-S3, based on the level calibration signals distributed on the two dynamic coordinate systems in the same time period, qualitative integration analysis and processing are performed, and a positive matching signal, a reverse matching signal and a non-matching signal are generated.

[0015] Furthermore, the specific operation steps of a type of model analysis and processing are as follows:

[0016] Randomly capture any 10 consecutive running performance coefficients yux on the performance dynamic coordinates i, and perform mean analysis on it to obtain the mean operating performance coefficient Jyux, convert the mean operating performance coefficient Jyux into a performance comparison reference line, that is, Y=Jyux, and draw the performance comparison reference line Y=Jyux on the performance dynamic coordinate system;

[0017] When the performance coefficient yux i When the performance comparison reference line Y = Jyux is above or above it, a superior performance signal is generated. i When it is below the performance comparison reference line Y=Jyux, a poor operating performance signal is generated.

[0018] Furthermore, the specific operation steps of the second type model analysis and processing are as follows:

[0019] Set the read / write threshold coefficient Yu1, and convert the read / write threshold coefficient Yu1 into a read / write comparison reference line, that is, Y=Yu1, and draw the read / write comparison reference line Y=Yu1 on the read / write dynamic coordinate system;

[0020] When the read-write value dux i When the reading / writing comparison reference line Yu1 is above or above it, a signal of excellent reading / writing performance is generated. i When it is below the read / write comparison reference line Yu1, a signal of inferior read / write performance is generated.

[0021] Furthermore, the specific steps of qualitative integration analysis are as follows:

[0022] Obtain the grade calibration signals distributed on the performance dynamic coordinate system and the read-write dynamic coordinate system at the same unit time level. If the signals captured at the same time are all superior type signals, a positive matching signal is generated. If the signals captured at the same time are all inferior type signals, a reverse matching signal is generated. If the signals captured at the same time are signals of two different grade calibrations, a mismatch signal is generated.

[0023] Further, the specific steps for optimizing the fixed-file analysis and processing are as follows:

[0024] Q1: When receiving the forward matching signal, the reverse matching signal and the mismatching signal, the number and the number of matching signals of each type in the unit time level are counted and marked as SL1, SL2 and SL3 respectively. If SL1<SL2+SL3 is satisfied, a high-level optimization instruction is generated; if SL1=SL2+SL3 is satisfied, a mid-level optimization instruction is generated; if SL1>SL2+SL3 is satisfied, a low-level optimization instruction is generated;

[0025] Q2: Count the sum of the numbers of inferior grade calibration signals and superior grade calibration signals distributed on the performance dynamic coordinate system and the read-write dynamic coordinate system respectively, and calculate the first proportion value Zbx1 and the second proportion value Zbx2 according to the percentage of the sum of the numbers of inferior signals of each type to the total sum of the grades of each type;

[0026] Q3: Based on the low-level optimization instructions, if Zbx1>Zbx2 is satisfied, an optimized read and write N-level instruction is generated; if Zbx1≤Zbx2 is satisfied, an optimized processor N-level instruction and an optimized memory N-level instruction are generated; based on the intermediate optimization instructions, if Zbx1>Zbx2 is satisfied, an optimized read and write N+1-level instruction is generated; if Zbx1≤Zbx2 is satisfied, an optimized processor N+1-level instruction and an optimized memory N+1-level instruction are generated; based on the high-level optimization instructions, if Zbx1>Zbx2 is satisfied, an optimized read and write N+2-level level signal is generated; if Zbx1≤Zbx2 is satisfied, a processor optimized N+2-level instruction and a memory optimized N+2-level instruction are generated.

[0027] Further, the specific operation steps of configuration optimization analysis and processing are as follows:

[0028] When receiving the instruction of optimizing read and write N levels, optimizing processor N levels, and optimizing memory N levels, the IO device, CPU, and memory in the big data system configuration are expanded by N levels accordingly, and an optimization completion signal is generated after the expansion is completed;

[0029] When receiving the instruction to optimize read and write N+1 level, the instruction to optimize processor N+1 level, and the instruction to optimize memory N+1 level, the IO device, CPU, and memory in the big data system configuration are expanded to N+1 level accordingly, and an optimization completion signal is generated after the expansion is completed;

[0030] When receiving the instructions for optimizing read and write N+2 levels, optimizing processor N+2 levels, and optimizing memory N+2 levels, the IO device, CPU, and memory in the big data system configuration are expanded to N+2 levels, and an optimization completion signal is generated after the expansion is completed.

[0031] Further, the specific steps for optimizing the iterative training analysis process are as follows:

[0032] When the optimization completion signal is received, the process value che in the configuration characteristic parameter information of the big data system at the unit time level is obtained. i 、Memory value nec i And the read and write value dux i , and normalize it to obtain the first test coefficient bet i ;

[0033] Get the process value che in the configuration characteristic parameter information of the big data system at the unit time level after optimizationj 、Memory value nec j And read and write value dux j , where j is a positive integer greater than or equal to 1, and normalized analysis is performed to obtain the second test coefficient bet j * ;

[0034] Bet the first test coefficient i With the second test coefficient bet * j Compare the values. If the bet value is met, i ≥bet * j When the optimization is not completed, an invalid optimization signal is generated and sent to the display terminal with the text "The current big data system configuration has not achieved the best optimization effect and needs to be optimized again". i <bet * j , an effective optimization signal is generated and sent to the display terminal with the text "The current big data system configuration parameters have achieved the best optimization effect".

[0035] Compared with the prior art, the present invention has the following beneficial effects:

[0036] 1. The present invention uses the methods of symbol calibration, formulaic processing, establishment of a dynamic coordinate system, and signal comparison output to accurately analyze the configuration parameter status of the big data system from different levels, and uses the method of data signal crossover to monitor and analyze the configuration parameters of the big data system at different levels in real time, thereby improving the accuracy and real-time performance of the monitoring of the configuration parameters of the big data system, and also laying a foundation for the subsequent optimization of the configuration status of the big data system;

[0037] 2. The present invention realizes accurate optimization analysis of the configuration parameters of the big data system through data summation and comparison, signal output and classification instruction output, and performs accurate tuning processing of the big data system configuration by executing the optimization instructions item by item, thereby improving the efficiency and accuracy of the tuning of the configuration parameters of the big data system and further promoting the efficient development of the big data system;

[0038] 3. The present invention utilizes symbolic calibration, iterative call of data, formulaic analysis and numerical comparison output to optimize the configuration parameters of the optimized big data system at the unit time level, perform iterative training and analysis, thereby accurately judging and outputting the optimization status of the configuration parameters of the big data system, truly realizing the precise optimization of the configuration parameters of the big data system, and promoting the stable and efficient development of the big data system. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to facilitate understanding by those skilled in the art, the present invention is further described below in conjunction with the accompanying drawings;

[0040] Figure 1 This is the overall system block diagram of the present invention. DETAILED DESCRIPTION

[0041] The technical solution of the present invention will be clearly and completely described below in conjunction with the embodiments. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0042] Embodiment 1:

[0043] like Figure 1 As shown, a big data system configuration parameter monitoring and tuning system based on deep learning includes a monitoring optimization and analysis platform, wherein a server is provided inside the monitoring optimization and analysis platform, and the server is connected to a data acquisition unit, a monitoring and analysis unit, a grade quantitative unit, an optimization control unit, a tuning evaluation unit and a display terminal in communication;

[0044] The monitoring, optimization and analysis platform is used to monitor and optimize the configuration parameter information of the big data system, obtain the configuration characteristic parameter information of the big data system in real time through the data acquisition unit, and send it to the monitoring and analysis unit;

[0045] It should be noted that the configuration characteristic parameter information is used to represent a type of data information that can reflect the performance status of the big data system during operation, and the configuration characteristic parameter information includes process value, memory value and read / write value, wherein the process value refers to the data value of the number of programs that the CPU runs simultaneously per unit time. It should be noted that when the performance value of the process value is larger, it means that the CPU has better operating performance and stronger performance in processing data, which can reflect the superiority of the big data system configuration from one aspect;

[0046] The memory value refers to the data value of the running memory size in the big data system configuration. It should be noted that when the memory value is larger, the running performance of the big data system configuration is better, which can also reflect the superiority of the big data system configuration from one aspect. The read-write value refers to the data value of the length of time it takes to complete a read or write on the disk;

[0047] The monitoring and analysis unit is used to receive the configuration characteristic parameter information of the big data system, and perform status monitoring and analysis processing accordingly, and generate a forward matching signal, a reverse matching signal and a non-matching signal accordingly, and send them all to the level quantitative unit. The level quantitative unit performs optimization and fixed-gear analysis processing on each type of matching signal received, and generates each type of extended gear position instruction accordingly, and sends them all to the optimization and control unit;

[0048] The optimization and control unit is used to receive various types of extended gear instructions, and perform configuration optimization analysis and processing on the configuration of the big data system based on this, generate an optimization completion signal based on this, and send it to the tuning and evaluation unit. The tuning and evaluation unit is used to receive the optimization completion signal, and retrieve the configuration feature parameter information of the big data system at the unit time level after optimization based on this, and perform optimization iterative training analysis and processing based on this, generate effective optimization signals and invalid optimization signals based on this, and send them to the display terminal for display in the form of text description.

[0049] Embodiment 2:

[0050] like Figure 1 As shown, when the monitoring and analysis unit receives the configuration characteristic parameter information of the big data system, it performs status monitoring and analysis processing accordingly. The specific operation process is as follows:

[0051] Obtain the process value and memory value in the configuration characteristic parameter information of the unit time level big data system in real time, and mark them as che i and nec i , normalize the process value and memory value, according to the formula yux i =e1×che i +e1×nec i , and obtain the operating performance coefficient yux at each time point i , where e1 and e2 are weight factor coefficients of process value and memory value respectively, and e1>e2>0, and e1+e2=0.9202. It should be noted that the weight factor coefficient is used to balance the weight of each data in the formula calculation, so as to promote the accuracy of the calculation result. Where i is a positive integer greater than or equal to 1, and i represents the unit time level of the monitoring stage;

[0052] Randomly capture the operating performance coefficient yux of any period of time in the unit time level i The performance dynamic coordinate system is established based on the time as the horizontal axis and the operating performance coefficient as the vertical axis. The operating performance coefficient yux for a period of time is plotted by plotting points. i It is outlined on the performance dynamic coordinate system, and a type of model analysis and processing is performed accordingly. The specific operation process is as follows:

[0053] Randomly capture any 10 consecutive running performance coefficients yux on the performance dynamic coordinates i And perform mean analysis on it, according to the formula Jyux=(yux1+yux2+…+yux 10 )÷10, and obtain the mean operating performance coefficient Jyux, and convert the mean operating performance coefficient Jyux into a performance comparison reference line, that is, Y=Jyux, and draw the performance comparison reference line Y=Jyux on the performance dynamic coordinate system;

[0054] It should be noted that yux1 represents any 10 consecutive operating performance coefficients yux i The first running performance coefficient in , yux2 represents any 10 consecutive running performance coefficients yux i The second running performance coefficient in, and so on, yux 10 Indicates any 10 consecutive operating performance coefficients yux i The tenth operating performance coefficient in ;

[0055] When the performance coefficient yux i When the performance comparison reference line Y = Jyux is above or above it, a superior performance signal is generated. i When it is below the performance comparison reference line Y=Jyux, a poor performance signal is generated;

[0056] Real-time capture of the read and write values ​​in the configuration characteristic parameter information of the big data system in the same time period as above, and mark it as dux i , with time as the horizontal axis and the reading and writing value as the vertical axis, a dynamic reading and writing coordinate system is established based on this, and the reading and writing value dux over a period of time is plotted by drawing points. i It is outlined on the read-write dynamic coordinate system, and the second type of model analysis and processing is performed accordingly. The specific operation process is as follows:

[0057] Set the read / write threshold coefficient Yu1, and convert the read / write threshold coefficient Yu1 into a read / write comparison reference line, that is, Y=Yu1, and draw the read / write comparison reference line Y=Yu1 on the read / write dynamic coordinate system;

[0058] When the read-write value dux i When the reading / writing comparison reference line Yu1 is above or above it, a signal of excellent reading / writing performance is generated. i When it is below the reading / writing comparison reference line Yu1, a signal indicating inferior reading / writing performance is generated;

[0059] Based on the excellent operation performance signals, inferior operation performance signals, excellent read / write performance signals and inferior read / write performance signals distributed on the two dynamic coordinate systems in the same time period, qualitative integration analysis and processing are performed accordingly. The specific operation process is as follows:

[0060] Obtain the grade calibration signals distributed on the performance dynamic coordinate system and the read-write dynamic coordinate system at the same unit time level, and if the signals captured at the same time are excellent operation performance signals and excellent read-write performance signals, a forward matching signal is generated; if the signals captured at the same time are inferior operation performance signals and inferior read-write performance signals, a reverse matching signal is generated; if the signals captured at the same time are inferior operation performance signals and excellent read-write performance signals or excellent operation performance signals and inferior read-write performance signals, a mismatch signal is generated;

[0061] and sending the generated forward matching signal, reverse matching signal and mismatching signal to the grade quantitative unit;

[0062] When the grade quantitative unit receives the forward matching signal, the reverse matching signal and the mismatching signal, it performs optimization and fixed-file analysis and processing accordingly. The specific operation process is as follows:

[0063] The number and the number of matching signals of each type in the unit time level are counted and marked as SL1, SL2 and SL3 respectively. If SL1<SL2+SL3 is satisfied, a high-level optimization instruction is generated; if SL1=SL2+SL3 is satisfied, a medium-level optimization instruction is generated; if SL1>SL2+SL3 is satisfied, a low-level optimization instruction is generated;

[0064] The sum of the numbers of inferior grade calibration signals and superior grade calibration signals distributed on the performance dynamic coordinate system and the read-write dynamic coordinate system are counted respectively, and the first proportion value Zbx1 and the second proportion value Zbx2 are obtained according to the percentage of the sum of the numbers of inferior signals of each type to the sum of the grades of each type;

[0065] It should be noted that the first ratio = the sum of the number of inferior running signals ÷ (the sum of the number of inferior running signals + the sum of the number of superior running signals), and the second ratio = the sum of the number of inferior read and write signals ÷ (the sum of the number of inferior read and write signals + the sum of the number of superior read and write signals);

[0066] According to the low-level optimization instructions, if Zbx1>Zbx2 is satisfied, an optimized read and write N-file instruction is generated; if Zbx1≤Zbx2 is satisfied, an optimized processor N-file instruction and an optimized memory N-file instruction are generated;

[0067] According to the intermediate optimization instructions, if Zbx1>Zbx2 is satisfied, then the optimized read and write N+1 file instructions are generated; if Zbx1≤Zbx2 is satisfied, then the optimized processor N+1 file instructions and optimized memory N+1 file instructions are generated;

[0068] According to the advanced optimization instruction, if Zbx1>Zbx2 is satisfied, an optimized read and write N+2 level signal is generated; if Zbx1≤Zbx2 is satisfied, a processor optimization N+2 level instruction and a memory optimization N+2 level instruction are generated;

[0069] And all types of generated extended gear position instructions are sent to the optimization control unit;

[0070] It should be pointed out that the specific value of "N" in N level, N+1 level and N+2 level is set by the technical personnel in the specific process of tuning the configuration parameters of the big data system, and "N" can represent the specific number of CPUs, the specific memory size, or the number of input and output devices;

[0071] When the optimization control unit receives various types of expansion gear instructions, it performs configuration optimization analysis and processing on the configuration of the big data system accordingly. The specific operation process is as follows:

[0072] When receiving the instruction of optimizing read and write N levels, optimizing processor N levels, and optimizing memory N levels, the IO device, CPU, and memory in the big data system configuration are expanded by N levels accordingly, and an optimization completion signal is generated after the expansion is completed;

[0073] When receiving the instruction to optimize read and write N+1 level, the instruction to optimize processor N+1 level, and the instruction to optimize memory N+1 level, the IO device, CPU, and memory in the big data system configuration are expanded to N+1 level accordingly, and an optimization completion signal is generated after the expansion is completed;

[0074] When receiving the instruction to optimize read and write N+2 levels, the instruction to optimize processor N+2 levels, and the instruction to optimize memory N+2 levels, the IO device, CPU, and memory in the big data system configuration are expanded by N+2 levels accordingly, and an optimization completion signal is generated after the expansion is completed, and the generated optimization completion signal is sent to the tuning evaluation unit;

[0075] When the tuning evaluation unit receives the optimization completion signal, it retrieves the configuration characteristic parameter information of the big data system at the unit time level after optimization, and performs optimization iterative training analysis and processing accordingly. The specific operation process is as follows:

[0076] When the optimization completion signal is received, the process value che in the configuration characteristic parameter information of the big data system at the unit time level is obtained. i 、Memory value nec i And the read and write value dux i , and normalize it according to the formula bet i =f1×che i +f2×nec i+f3×dux i , find the first test coefficient bet i ;

[0077] Get the process value che in the configuration characteristic parameter information of the big data system at the unit time level after optimization j 、Memory value nec j And the read and write value dux j , where j is a positive integer greater than or equal to 1, and normalized analysis is performed according to the formula bet j * =f1×che j +f2×nec j +f3×dux j , find the second test coefficient bet j * , where f1, f2 and f3 are the correction factor coefficients of process value, memory value and read / write value respectively, and f3>f1>f2>0, f1+f2+f3=5.0301;

[0078] It should be noted that the correction factor coefficient is used to correct the deviation of various parameters in the process of formula calculation, so as to make the calculation more accurate and the parameter data;

[0079] It should also be noted that j represents the unit time level in the post-optimization stage, the time interval of the i unit time level is equal to the time interval of the j unit time level, i represents the operation time interval of the big data system without optimizing the big data system configuration, and j represents the operation time interval of the big data system after the big data system configuration has been optimized accordingly;

[0080] Bet the first test coefficient i With the second test coefficient bet * j Compare the values. If the bet value is met, i ≥bet * j When the optimization is not completed, an invalid optimization signal is generated and sent to the display terminal with the text "The current big data system configuration has not achieved the best optimization effect and needs to be optimized again". i <bet * j , an effective optimization signal is generated and sent to the display terminal with the text "The current big data system configuration parameters have achieved the best optimization effect".

[0081] The above formulas are obtained by collecting a large amount of data and performing software simulation, and a formula close to the actual value is selected. The coefficients in the formula are set by technicians in this field according to actual conditions;

[0082] Such as the formula: yux i =e1×che i +e1×nec i ;

[0083] A technician in this field collects multiple groups of sample data and sets corresponding weight factor coefficients for each group of sample data; substitutes the set weight factor coefficients and the collected sample data into the formula, any two formulas constitute a binary linear equation system, the calculated coefficients are screened and averaged, and the values ​​of e1 and e2 are obtained to be 0.2325 and 0.6877 respectively;

[0084] The size of the coefficient is to quantify each parameter to obtain a specific value for subsequent comparison. The size of the coefficient depends on the amount of sample data and the preliminary setting of the corresponding weight factor coefficient for each group of sample data by technical personnel in this field; as long as it does not affect the proportional relationship between the parameter and the quantized value.

[0085] When the present invention is used, the process value and memory value of the configuration parameters of the big data system are collected in real time, and the operation performance coefficient of the configuration parameters of the big data system is obtained by means of symbol calibration and normalization processing, and a performance dynamic coordinate system is established by means of point plotting based on the operation performance coefficient after data processing, and an operation performance superior signal and an operation performance inferior signal for determining the state of the configuration parameters of the big data system are obtained by means of selecting any interval points, setting reference lines, and outputting signalized comparisons;

[0086] By collecting the read and write values ​​of the configuration parameters of the big data system in real time and establishing a dynamic coordinate system for reading and writing, the read and write performance superior signals and read and write performance inferior signals for determining the status of the configuration parameters of the big data system are obtained through the setting of threshold coefficients, the setting of reference lines, and the signaled comparison output;

[0087] By using the integrated analysis method of cross-data signal, the two types of performance signals are integrated and analyzed, thereby realizing real-time monitoring and analysis of the configuration parameters of the big data system. By monitoring and analyzing the configuration parameters of the big data system at different levels, the accuracy and real-time performance of the monitoring of the configuration parameters of the big data system are improved. While realizing accurate judgment and analysis of the configuration parameter status of the big data system, it also lays the foundation for the subsequent optimization of the configuration status of the big data system.

[0088] Based on the configuration parameter status judgment results of the big data system, accurate optimization analysis of the configuration parameters of the big data system is achieved through data summation comparison, signal output and classification instruction output. The configuration of the big data system is accurately tuned by executing the optimization instructions item by item, thereby improving the efficiency and accuracy of the tuning of the configuration parameters of the big data system and further promoting the efficient development of the big data system.

[0089] By utilizing symbolic calibration, iterative data calls, formulaic analysis, and numerical comparison output, the configuration parameters of the optimized big data system at the unit time level are optimized, iteratively trained, analyzed, and processed, thereby accurately judging and outputting the optimization status of the big data system configuration parameters, truly achieving precise optimization of the big data system configuration parameters and promoting the stable and efficient development of the big data system.

[0090] The preferred embodiments of the present invention disclosed above are only used to help explain the present invention. The preferred embodiments do not describe all the details in detail, nor do they limit the invention to only specific implementation methods. Obviously, many modifications and changes can be made according to the content of this specification. This specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the present invention, so that those skilled in the art can understand and use the present invention well. The present invention is limited only by the claims and their full scope and equivalents.

Claims

1. A big data system configuration parameter monitoring and tuning system based on deep learning, including a monitoring optimization and analysis platform, characterized in that: The monitoring, optimization and analysis platform is internally provided with a server, and the server is connected to a data acquisition unit, a monitoring and analysis unit, a grade quantitative unit, an optimization and control unit, a tuning and evaluation unit and a display terminal; The monitoring, optimization and analysis platform is used to monitor and optimize the configuration parameter information of the big data system. The configuration characteristic parameter information of the big data system is obtained in real time through the data acquisition unit, and sent to the monitoring and analysis unit. The monitoring and analysis unit is used to receive the configuration characteristic parameter information of the big data system, and perform state monitoring and analysis processing based on it, and generate forward matching signals, reverse matching signals and non-matching signals based on it, and send them all to the level quantitative unit. The level quantitative unit performs optimization and fixed-gear analysis processing on each type of matching signal received, and generates each type of extended gear position instruction based on it, and sends them all to the optimization and control unit. The optimization control unit is used to receive various types of extended gear position instructions, and perform configuration optimization analysis and processing on the configuration of the big data system accordingly, and generate an optimization completion signal accordingly, and send it to the tuning evaluation unit. The tuning evaluation unit is used to receive the optimization completion signal, and retrieve the configuration characteristic parameter information of the big data system at the unit time level after optimization accordingly, and perform optimization iterative training analysis and processing accordingly, and generate effective optimization signals and invalid optimization signals accordingly, and send them to the display terminal in the form of text description for display; The specific steps of condition monitoring analysis and processing are as follows: S1: Real-time acquisition of the process value in the configuration characteristic parameter information of the unit time level big data system che i and memory value nec i , where i is a positive integer greater than or equal to 1, the process value and memory value are normalized to obtain the running performance coefficient yux at each time point i ; S2: According to step S1, randomly capture the operating performance coefficient yux of any period of time in the unit time level i The performance dynamic coordinate system is established based on the time as the horizontal axis and the operating performance coefficient as the vertical axis. The operating performance coefficient yux for a period of time is plotted by plotting points. i Outlined on a performance dynamic coordinate system, and based on this, a type of model analysis processing is performed, and a superior operating performance signal and a inferior operating performance signal are generated; S3: Real-time capture of the read and write value dux in the configuration characteristic parameter information of the big data system in the same time period as step S2 i , with time as the horizontal axis and the reading and writing value as the vertical axis, a dynamic reading and writing coordinate system is established based on this, and the reading and writing value dux over a period of time is plotted by drawing points. i Outlined on the read-write dynamic coordinate system, and based on this, perform two-type model analysis and processing, and generate a signal of superior read-write performance and a signal of inferior read-write performance; S4: According to steps S2-S3, based on the level calibration signals distributed on the two dynamic coordinate systems in the same time period, qualitative integration analysis and processing are performed, and a positive matching signal, a reverse matching signal and a non-matching signal are generated; The specific steps of a type of model analysis and processing are as follows: Randomly capture any 10 consecutive running performance coefficients yux on the performance dynamic coordinates i , and perform mean analysis on it to obtain the mean operating performance coefficient Jyux, convert the mean operating performance coefficient Jyux into a performance comparison reference line, that is, Y=Jyux, and draw the performance comparison reference line Y=Jyux on the performance dynamic coordinate system; When the performance coefficient yux i When the performance comparison reference line Y = Jyux is above or above it, a superior performance signal is generated. i When it is below the performance comparison reference line Y=Jyux, a poor performance signal is generated; The specific steps of the second type of model analysis and processing are as follows: Set the read / write threshold coefficient Yu1, and convert the read / write threshold coefficient Yu1 into a read / write comparison reference line, that is, Y=Yu1, and draw the read / write comparison reference line Y=Yu1 on the read / write dynamic coordinate system; When the read-write value dux i When the reading / writing comparison reference line Yu1 is above or above it, a signal of excellent reading / writing performance is generated. i When it is below the reading / writing comparison reference line Yu1, a signal indicating inferior reading / writing performance is generated; The specific steps of qualitative integrated analysis are as follows: Obtain the grade calibration signals distributed on the performance dynamic coordinate system and the read-write dynamic coordinate system at the same unit time level. If the signals captured at the same time are all superior type signals, a positive matching signal is generated. If the signals captured at the same time are all inferior type signals, a reverse matching signal is generated. If the signals captured at the same time are signals of two different grade calibrations, a mismatch signal is generated. The specific steps for optimizing the fixed-file analysis and processing are as follows: Q1: When receiving the forward matching signal, the reverse matching signal and the mismatching signal, the number and the number of matching signals of each type in the unit time level are counted and marked as SL1, SL2 and SL3 respectively. If SL1<SL2+SL3 is satisfied, a high-level optimization instruction is generated; if SL1=SL2+SL3 is satisfied, a mid-level optimization instruction is generated; if SL1>SL2+SL3 is satisfied, a low-level optimization instruction is generated; Q2: Count the sum of the numbers of inferior grade calibration signals and superior grade calibration signals distributed on the performance dynamic coordinate system and the read-write dynamic coordinate system respectively, and calculate the first proportion value Zbx1 and the second proportion value Zbx2 according to the percentage of the sum of the numbers of inferior signals of each type to the total sum of the grades of each type; Q3: According to the low-level optimization instructions, if Zbx1>Zbx2 is satisfied, an optimized read and write N-grade instruction is generated; if Zbx1≤Zbx2 is satisfied, an optimized processor N-grade instruction and an optimized memory N-grade instruction are generated; according to the intermediate optimization instructions, if Zbx1>Zbx2 is satisfied, an optimized read and write N+1-grade instruction is generated; if Zbx1≤Zbx2 is satisfied, an optimized processor N+1-grade instruction and an optimized memory N+1-grade instruction are generated; according to the high-level optimization instructions, if Zbx1>Zbx2 is satisfied, an optimized read and write N+2-grade level signal is generated; if Zbx1≤Zbx2 is satisfied, a processor optimized N+2-grade instruction and a memory optimized N+2-grade instruction are generated; The specific steps for configuration optimization analysis and processing are as follows: When receiving the instruction of optimizing read and write N levels, optimizing processor N levels, and optimizing memory N levels, the IO device, CPU, and memory in the big data system configuration are expanded by N levels accordingly, and an optimization completion signal is generated after the expansion is completed; When receiving the instruction to optimize read and write N+1 level, the instruction to optimize processor N+1 level, and the instruction to optimize memory N+1 level, the IO device, CPU, and memory in the big data system configuration are expanded to N+1 level accordingly, and an optimization completion signal is generated after the expansion is completed; When receiving the instruction to optimize read and write N+2 levels, the instruction to optimize processor N+2 levels, and the instruction to optimize memory N+2 levels, the IO device, CPU, and memory in the big data system configuration are expanded to N+2 levels accordingly, and an optimization completion signal is generated after the expansion is completed; The specific steps for optimizing iterative training analysis and processing are as follows: When the optimization completion signal is received, the process value che in the configuration characteristic parameter information of the big data system at the unit time level is obtained. i 、Memory value nec i And read and write value dux i , and normalize it to obtain the first test coefficient bet i ; Get the process value che in the configuration characteristic parameter information of the big data system at the unit time level after optimization j 、Memory value nec j And the read and write value dux j , where j is a positive integer greater than or equal to 1, and it is normalized for analysis; Bet the first test coefficient i With the second test coefficient bet * j Compare the values. If the bet value is met, i ≥bet * j When the optimization is not completed, an invalid optimization signal is generated and sent to the display terminal with the text "The current big data system configuration has not achieved the best optimization effect and needs to be optimized again". i <bet * j , an effective optimization signal is generated and sent to the display terminal with the text "The current big data system configuration parameters have achieved the best optimization effect".

Images