A GIL health status monitoring system and method based on big data

Abstract

The present invention discloses a GIL health status monitoring system and method based on big data, belonging to the technical field of data analysis. The present invention collects the status data of the CPython interpreter before and after the work efficiency of the CPython interpreter decreases in history, generates a visual data set of the CPython interpreter; collects the status data of the CPython interpreter when the GIL is in an unhealthy state in history, generates a GIL feature data set; calculates the CPython interpreter work efficiency threshold and the GIL health status threshold respectively; performs real-time monitoring and early warning on the CPython interpreter according to the work efficiency threshold; calculates the real-time health index of the GIL after the early warning; judges the real-time health index of the GIL according to the GIL health status threshold, and performs a GIL health early warning.

Description

Technical Field

[0001] The present invention relates to the technical field of data analysis, and in particular to a GIL health status monitoring system and method based on big data. Background Art

[0002] The design of the Python language began in 1989, when it was written by Guido van Rossum in the Netherlands. The earliest Python interpreter was written in C and was first released in 1991 as Python 0.9.0. This interpreter was called CPython, where the "C" stands for being written in C. The Python language evolved rapidly over the next few years, introducing many new features and functionality. As the user base grew, the demand for interpreter performance and functionality also increased. The CPython interpreter is constantly evolving, adding new features, optimizing performance, improving language features, and improving the standard library. The developer community actively solves problems, proposes improvements, and releases new Python versions, including the popular 3.x version series. The early CPython interpreter introduced the GIL to simplify memory management. The design idea at the time was to use reference counting to manage object lifecycles, and the GIL could avoid race conditions when multiple threads operate on shared data. In the development of the current CPython interpreter, GIL is often used, but GIL may also be in an unhealthy state, and the unhealthy GIL will lead to a decrease in the working efficiency of the CPython interpreter, so the health of GIL is crucial; however, because the reasons for the decrease in the working efficiency of the CPython interpreter are not only the health of the GIL, it is very important to accurately predict the working efficiency of the CPython interpreter and accurately find out whether the efficiency decrease is caused by the unhealthy state of GIL; based on the above problems, the present invention provides a GIL health status monitoring system and method based on big data. Summary of the invention

[0003] The purpose of the present invention is to provide a GIL health status monitoring system and method based on big data to solve the problems raised in the above background technology.

[0004] In order to solve the above technical problems, the present invention provides the following technical solutions:

[0005] A GIL health status monitoring method based on big data, the method comprising the following steps:

[0006] S101. Collect the E-time working time and workload of the CPython interpreter in history, and calculate the working efficiency of the CPython interpreter. The formula is: In the formula, G is the total amount of work, D is the working time, and then the average value of the work efficiency for E times is calculated. When the work efficiency is lower than the average value of α, it is determined that the work efficiency has decreased.

[0007] S102. Collect the state data of the CPython interpreter before and after the decrease in work efficiency in history. According to the working time period of the CPython interpreter, the state data in the same time is classified as the same kind of state data. Suppose the state data of the CPython interpreter before and after the decrease in work efficiency collected are {q1, q2, q3... q n}, {q’1, q’2, q’3... q’ n}, where q1, q2, q3... q n are the 1st, 2nd, 3rd... nth kinds of state data before the decrease in the work efficiency of the CPython interpreter, and q’1, q’2, q’3... q’ n are the 1st, 2nd, 3rd... nth kinds of state data after the decrease in the work efficiency of the CPython interpreter, and n is a positive integer.

[0008] S103. Compare the n kinds of state data of the CPython interpreter before and after the decrease in work efficiency, and calculate the difference of each kind of state data before and after the decrease in work efficiency respectively. The formula is:

[0009] C = |s i - s’ i |

[0010] In the above formula, C is the difference of each kind of state data before and after the decrease in work efficiency, s i is the value of the state data before the decrease in work efficiency, s’ i is the value of the state data after the decrease in work efficiency, 0 < i < n, and i is a positive integer.

[0011] S104. After calculating the difference of each kind of state data before and after the decrease in work efficiency, n kinds of difference data {C1, C2, C3... C n} are obtained. C1, C2, C3... C n are the differences of the 1st, 2nd, 3rd... nth kinds of state data calculated, and n is a positive integer. Calculate the standard deviation of the differences of n kinds of state data. The formula is:

[0012]

[0013] In the above formula, C_b is the standard deviation of the differences of n kinds of state data, C_p is the average value of the differences of n kinds of state data, 0 < i < n;

[0014] S105, judging the visual data that affects the working efficiency of the CPython interpreter according to the calculated standard deviation of the difference values ​​of the n kinds of state data, specifically: when C i >C_b, judge C i The corresponding status data is the visual data L that affects the efficiency of the CPython interpreter. i ≤C_b, judge C i The corresponding state data is not visible data that affects the working efficiency of the CPython interpreter, so it is discarded; after judging n kinds of state data in turn, the visible data set that affects the working efficiency of the CPython interpreter is {L1, L2, L3...L m}, L1, L2, L3...L m The first, second, third, ...m types of visual data that affect the efficiency of the CPython interpreter are determined, where m is a positive integer, 1 <m<n。

[0015] By using historical data to compare and analyze various status data before and after the CPython interpreter's work efficiency decreases, we can obtain status data that can reflect the decrease in work efficiency and regard it as visual data. By using visual data, we can clearly monitor the efficiency of the CPython interpreter in real-time work and predict the work efficiency.

[0016] S200, collecting historical work data of the CPython interpreter when the GIL is in an unhealthy state, analyzing and classifying the work data, and searching and generating a feature data set of the GIL;

[0017] Furthermore, the specific steps to find and generate the feature data set of GIL are:

[0018] S201, respectively collect the status data of the CPython interpreter when the GIL is in an unhealthy state and a healthy state in history, and classify the status data at the same time as the same status data according to the working time period of the CPython interpreter. Suppose the status data of the CPython interpreter when the GIL is in a healthy state and an unhealthy state are {p1, p2, p3...p n}, {p'1, p'2, p'3...p' n}, p1, p2, p3...p n The collected data are the 1st, 2nd, 3rd, ... nth states of the CPython interpreter when the GIL is in a healthy state, p'1, ​​p'2, p'3, ... p' n The collected CPython interpreter status data of 1st, 2nd, 3rd...nth when GIL is in an unhealthy state, where n is a positive integer;

[0019] S202. Compare the collected status data p of the CPython interpreter when the GIL is in a healthy state and an unhealthy state. i and p' i ,After comparison, the state data types that change when GIL is in different states are used as the ,associated data sets of GIL;

[0020] S203, after obtaining the GIL related data by comparison, compare the same type of state data when the GIL is in a healthy state and the same type of state data when the GIL is in an unhealthy state, extract the data types of the state data that has changed while the GIL state has not changed but the CPython interpreter is working, and sort out the extracted data types to obtain the GIL disturbance data set; in the associated data obtained by searching, although the state data has changed, it is not necessarily caused by the unhealthy GIL, therefore, by comparing the state data of the GIL in the same state, the change data that is not related to the GIL state can be found, and after removing the irrelevant change data in the associated data, the state data that can reflect the GIL state when the CPython interpreter is working is obtained;

[0021] S204: Compare the associated data set of the GIL with the disturbed data set, remove the disturbed data in the associated data set, and use the removed data set as the feature data set of the GIL.

[0022] S300, analyzing the historical data of the CPython interpreter when it was working, and calculating the CPython interpreter working efficiency threshold and the GIL health status threshold respectively;

[0023] Furthermore, the specific steps for calculating the CPython interpreter efficiency threshold and the GIL health status threshold are as follows:

[0024] S301, calculate the average value s_p of each visual data before the efficiency decreases in the history of the CPython interpreter, collect the data difference of each state data in the visual data set before and after the h times of the CPython interpreter's work efficiency decreases in the history, and calculate the average value of each visual data difference, the formula is:

[0025]

[0026] In the formula, C_p is the average value of the difference before and after the work efficiency of each visual data decreases, c_L is the difference of each visual data before and after the work efficiency decreases collected h times, and h is the number of the difference of each visual data; finally, the average value of the difference before and after the work efficiency decreases of m kinds of visual data is calculated;

[0027] S302, collecting the average number a of changes in visible data when the working efficiency of the CPython interpreter decreases in history, and calculating the working efficiency threshold of the CPython interpreter, the formula is:

[0028]

[0029] In the formula, e_y represents the efficiency threshold of the CPython interpreter, a represents the average number of changes in the visual data when the efficiency of the CPython interpreter decreases in history, C_p represents the average difference between each visual data before and after the efficiency decreases, and m is the total number of visual data;

[0030] For the calculation of the CPython interpreter work efficiency threshold, the sum of the changes in the visible data when the work efficiency decreases in the history is used, and then multiplied by the ratio of the average number of state data changed each time to the total number of state data, which can reflect the standard of the CPython interpreter work efficiency decrease;

[0031] S303, collect the total number of status data that changes after the CPython interpreter's working efficiency decreases when the GIL is in an unhealthy state, and calculate the GIL health state threshold. The formula is:

[0032]

[0033] In the formula, he is the GIL health status threshold, and b is the number of GIL feature data in the status data that changes after the CPython interpreter's efficiency decreases.

[0034] S400, collecting a visual data set in real time when the CPython interpreter is working, and performing real-time monitoring and early warning on the CPython interpreter according to a work efficiency threshold;

[0035] Furthermore, the specific steps for real-time monitoring and early warning of the CPython interpreter according to the work efficiency threshold are as follows:

[0036] S401, collect visual data of the CPython interpreter at work in real time, and calculate a real-time efficiency index based on the collected visual data, the formula is:

[0037]

[0038] In the formula, e_s is the efficiency index of the CPython interpreter at work, m is the number of visible data, and ss is r The real-time data value of each visual data, s_p is the average value of each visual data before the efficiency decreases in history, 1 <r<m;

[0039] S402. When e_s≥e_y, it is predicted that the working efficiency of the CPython interpreter will decrease, and an efficiency warning is issued. When e_s>e_y, it is predicted that the working efficiency of the CPython interpreter will not decrease.

[0040] S500, after providing an efficiency warning to the CPython interpreter, calculating a real-time health index of the GIL according to the real-time visual data and GIL feature data;

[0041] Furthermore, the specific steps of calculating the real-time health index of the GIL based on the real-time visual data and the GIL characteristic data are as follows:

[0042] S501, after issuing an efficiency warning to the CPython interpreter, analyzing the visual data, respectively calculating the difference between the collected real-time visual data and the average value of each visual data before the efficiency decreases in history, and when the difference is greater than 0, determining that the corresponding visual data is status data that affects the decrease in the working efficiency of the CPython interpreter;

[0043] S502, after calculation, the state data that affects the decrease in the working efficiency of the CPython interpreter in the collected real-time visual data is {g1, g2, g3...g j}, g1, g2, g3...g j The first, second, third, ...jth state data that affect the efficiency of the CPython interpreter are determined, where j is a positive integer, 1 <j<m;

[0044] S503, using the GIL feature data set to filter the data that affects the decline in the efficiency of the CPython interpreter, extracting the state data of the GIL feature data in the set in S502 as {w1, w2, w3...w f}, w1, w2, w3...w f is the characteristic data of the 1st, 2nd, 3rd...fth GIL in the data set that affects the decline in the efficiency of the CPython interpreter, where f is a positive integer;

[0045] S504. Calculate the real-time GIL health index using the formula:

[0046]

[0047] In the formula, he_s is the real-time GIL health index, f is the number of GIL feature data among the j status data that affect the decline in CPython interpreter efficiency, j is the total number of status data that affect the decline in CPython interpreter efficiency, sw esg is the real-time data value of f GIL feature data among the state data that affects the decrease in the working efficiency of the CPython interpreter. u is the data value of j state data that affects the decrease in the working efficiency of the CPython interpreter, where 1 < e < f and 1 < u < j.

[0048] When the CPython interpreter is working and a decrease in efficiency is detected, it is not possible to simply determine that the GIL is in an unhealthy state based on the decrease in efficiency because there are various factors that can cause the working efficiency of the CPython interpreter to decrease. Therefore, after issuing an efficiency warning, it is also necessary to calculate the health index of the GIL to determine whether the decrease in the working efficiency of the CPython interpreter is caused by an unhealthy GIL. If so, a warning is issued; if not, the possibility of the GIL being in an unhealthy state is excluded, and other reasons for the decrease in the working efficiency of the CPython interpreter are searched for.

[0049] S600. Determine whether the real-time health index of the GIL is in an unhealthy state based on the GIL health status threshold and issue a GIL health warning.

[0050] Furthermore, the specific steps for issuing a GIL health warning are as follows:

[0051] Use the GIL health status threshold to judge the GIL health index calculated in real time. When he_s ≥ he_y, judge that the GIL is in an unhealthy state and issue a warning; when he_s < he_y, judge that the GIL is in a healthy state and exclude the possibility that the decrease in the working efficiency of the CPython interpreter is caused by an unhealthy GIL.

[0052] A GIL health status monitoring system based on big data, where the GIL health status monitoring system includes a data collection module, a data analysis module, a threshold calculation module, a working efficiency warning module, and a GIL status warning module;

[0053] The data collection module is used to collect the working state data and GIL health status data of the CPython interpreter in history;

[0054] The data analysis module is used to analyze the historical data collected and calculate the visible data set of the decrease in the working efficiency of the CPython interpreter and the feature data set of the GIL;

[0055] The threshold calculation module is used to calculate the working efficiency threshold of the CPython interpreter and the GIL health status threshold;

[0056] The working efficiency warning module is used to judge the real-time working efficiency of the CPython interpreter according to the working efficiency threshold and issue a warning;

[0057] The GIL status warning module is used to judge the real-time GIL status according to the GIL health status threshold and issue a warning.

[0058] The data analysis module includes a visual data calculation unit and a GIL feature data calculation unit;

[0059] The visual data calculation unit is used to calculate the state data before and after the work efficiency of the CPython interpreter in history decreases, obtain the difference before and after the work efficiency decreases for each state data, judge the difference of each state data, and obtain the visual data set of the work efficiency decrease;

[0060] The GIL characteristic data calculation unit is used to collect and judge the status data of the CPython interpreter when the GIL is healthy and unhealthy in history, and analyze the status data of the GIL in different states and the same state respectively to obtain the characteristic data set of the GIL.

[0061] The threshold calculation module includes a work efficiency threshold calculation unit and a GIL health status threshold calculation unit;

[0062] The work efficiency threshold calculation unit is used to perform difference calculation on the visual data in the work of the CPython interpreter in history, and calculate the work efficiency threshold of the CPython interpreter by the average value and number of the difference of the visual data in history;

[0063] The GIL health status threshold calculation unit is used to calculate the proportion of the number of GILs in an unhealthy state in the total number of changed data when the working efficiency of the CPython interpreter decreases in history, and obtain the GIL health status threshold.

[0064] Compared with the prior art, the beneficial effects achieved by the present invention are:

[0065] 1. The present invention collects and analyzes the historical working data of the CPython interpreter, and compares and obtains visual data reflecting the working efficiency of the CPython interpreter. The working efficiency of the CPython interpreter can be predicted and early warning can be carried out through real-time monitoring of the visual data, thereby ensuring the working efficiency of the CPython interpreter.

[0066] 2. The present invention searches for characteristic data of GIL in the CPython interpreter, and uses the searched characteristic data to realize real-time monitoring of the health status of GIL. When it is determined that GIL is in an unhealthy state, an early warning is issued, and GIL is processed, thereby reducing the working time of the CPython interpreter.

[0067] 3. After the present invention issues an early warning for the decline in the working efficiency of the CPython interpreter, it determines whether the cause of the decline in the working efficiency of the CPython interpreter is due to an unhealthy GIL. The cause of the decline in the working efficiency of the CPython interpreter can be accurately investigated, and a preliminary judgment can be made on the health status of the GIL based on the working efficiency of the CPython interpreter to avoid unnecessary resource consumption. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0069] Figure 1 It is a module distribution diagram of a GIL health status monitoring system based on big data of the present invention;

[0070] Figure 2 It is a schematic diagram of the steps of a GIL health status monitoring method based on big data of the present invention. DETAILED DESCRIPTION

[0071] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0072] See also Figure 1-Figure 2 , the present invention provides a technical solution:

[0073] A GIL health status monitoring method based on big data, the method comprising the following steps:

[0074] S101. Collect the E-time working time and workload of the CPython interpreter in history, and calculate the working efficiency of the CPython interpreter. The formula is: In the formula, G is the total amount of work, D is the working time, and then the average of the work efficiency of E times is calculated. It is judged that when the work efficiency is lower than the average value of α, the work efficiency decreases;

[0075] S102, collecting the status data of the CPython interpreter before and after the work efficiency decreases in history, and classifying the status data at the same time as the same type of status data according to the working time period of the CPython interpreter, and assuming that the collected status data of the CPython interpreter before and after the work efficiency decreases are [q1, q2, q3...q n}, {q'1, q'2, q'3...q'n}, q1, q2, q3... q n are the first, second, third... nth state data before the working efficiency of the CPython interpreter decreases, and q’1, q’2, q’3... q’ n are the first, second, third... nth state data after the working efficiency of the CPython interpreter decreases, where n is a positive integer;

[0076] S103. Compare the n state data before and after the working efficiency of the CPython interpreter decreases, and calculate the difference of each state data before and after the working efficiency decreases respectively. The formula is:

[0077] C = |s i - s’ i |

[0078] In the above formula, C is the difference of each state data before and after the working efficiency decreases, s i is the value of the state data before the working efficiency decreases, and s’ i is the value of the state data after the working efficiency decreases, where 0 < i < n and i is a positive integer;

[0079] S104. After calculating the difference of each state data before and after the working efficiency decreases, n difference data {C1, C2, C3... C n} are obtained. C1, C2, C3... C n are the differences of the first, second, third... nth state data calculated, where n is a positive integer. Calculate the standard deviation of the differences of the n state data. The formula is:

[0080]

[0081] In the above formula, C_b is the standard deviation of the differences of the n state data, C_p is the average value of the differences of the n state data, and 0 < i < n;

[0082] S105. Judge the visible data affecting the working efficiency of the CPython interpreter according to the standard deviation of the differences of the n state data calculated. Specifically: when C i > C_b, judge that the state data corresponding to C i is the visible data L affecting the working efficiency of the CPython interpreter. When C i ≤ C_b, judge that the state data corresponding to C i is not the visible data affecting the working efficiency of the CPython interpreter and discard it; After judging the n state data in turn, the visible data set affecting the decrease of the working efficiency of the CPython interpreter is {L1, L2, L3... L m}, L1, L2, L3... L mThe first, second, third, ...m types of visual data that affect the efficiency of the CPython interpreter are determined, where m is a positive integer, 1 <m<n。

[0083] By using historical data to compare and analyze various status data before and after the CPython interpreter's work efficiency decreases, we can obtain status data that can reflect the decrease in work efficiency and regard it as visual data. By using visual data, we can clearly monitor the efficiency of the CPython interpreter in real-time work and predict the work efficiency.

[0084] S200, collecting historical work data of the CPython interpreter when the GIL is in an unhealthy state, analyzing and classifying the work data, and searching and generating a feature data set of the GIL;

[0085] The specific steps to find and generate the GIL feature data set are:

[0086] S201, respectively collect the status data of the CPython interpreter when the GIL is in an unhealthy state and a healthy state in history, and classify the status data at the same time as the same status data according to the working time period of the CPython interpreter. Suppose the status data of the CPython interpreter when the GIL is in a healthy state and an unhealthy state are {p1, p2, p3...p n}, {p'1, p'2, p'3...p' n}, p1, p2, p3...p n The collected data are the 1st, 2nd, 3rd, ... nth states of the CPython interpreter when the GIL is in a healthy state, p'1, ​​p'2, p'3, ... p' n The collected CPython interpreter status data of 1st, 2nd, 3rd...nth when GIL is in an unhealthy state, where n is a positive integer;

[0087] S202. Compare the collected status data p of the CPython interpreter when the GIL is in a healthy state and an unhealthy state. i and p' i ,After comparison, the state data types that change when GIL is in different states are used as the ,associated data sets of GIL;

[0088] S203, after obtaining the GIL related data by comparison, compare the same type of state data when the GIL is in a healthy state and the same type of state data when the GIL is in an unhealthy state, extract the data types of the state data that has changed while the GIL state has not changed but the CPython interpreter is working, and sort out the extracted data types to obtain the GIL disturbance data set; in the associated data obtained by searching, although the state data has changed, it is not necessarily caused by the unhealthy GIL, therefore, by comparing the state data of the GIL in the same state, the change data that is not related to the GIL state can be found, and after removing the irrelevant change data in the associated data, the state data that can reflect the GIL state when the CPython interpreter is working is obtained;

[0089] S204: Compare the associated data set of the GIL with the disturbed data set, remove the disturbed data in the associated data set, and use the removed data set as the feature data set of the GIL.

[0090] S300, analyzing the historical data of the CPython interpreter when it was working, and calculating the CPython interpreter working efficiency threshold and the GIL health status threshold respectively;

[0091] The specific steps to calculate the CPython interpreter efficiency threshold and the GIL health status threshold are:

[0092] S301, calculate the average value s_p of each visual data before the efficiency decreases in the history of the CPython interpreter, collect the data difference of each state data in the visual data set before and after the h times of the CPython interpreter's work efficiency decreases in the history, and calculate the average value of each visual data difference, the formula is:

[0093]

[0094] In the formula, C_p is the average value of the difference before and after the work efficiency of each visual data decreases, c_L is the difference of each visual data before and after the work efficiency decreases collected h times, and h is the number of the difference of each visual data; finally, the average value of the difference before and after the work efficiency decreases of m kinds of visual data is calculated;

[0095] S302, collecting the average number a of changes in visible data when the working efficiency of the CPython interpreter decreases in history, and calculating the working efficiency threshold of the CPython interpreter, the formula is:

[0096]

[0097] In the formula, e_y represents the efficiency threshold of the CPython interpreter, a represents the average number of changes in the visual data when the efficiency of the CPython interpreter decreases in history, C_p represents the average difference between each visual data before and after the efficiency decreases, and m is the total number of visual data;

[0098] For the calculation of the CPython interpreter work efficiency threshold, the sum of the changes in the visible data when the work efficiency decreases in the history is used, and then multiplied by the ratio of the average number of state data changed each time to the total number of state data, which can reflect the standard of the CPython interpreter work efficiency decrease;

[0099] S303, collect the total number of status data that changes after the CPython interpreter's working efficiency decreases when the GIL is in an unhealthy state, and calculate the GIL health state threshold. The formula is:

[0100]

[0101] In the formula, he is the GIL health status threshold, and b is the number of GIL feature data in the status data that changes after the CPython interpreter's efficiency decreases.

[0102] S400, collecting a visual data set in real time when the CPython interpreter is working, and performing real-time monitoring and early warning on the CPython interpreter according to a work efficiency threshold;

[0103] The specific steps for real-time monitoring and early warning of the CPython interpreter based on the work efficiency threshold are:

[0104] S401, collect visual data of the CPython interpreter at work in real time, and calculate a real-time efficiency index based on the collected visual data, the formula is:

[0105]

[0106] In the formula, e_s is the efficiency index of the CPython interpreter at work, m is the number of visible data, and ss is r The real-time data value of each visual data, s_p is the average value of each visual data before the efficiency decreases in history, 1 <r<m;

[0107] S402. When e_s≥e_y, it is predicted that the working efficiency of the CPython interpreter will decrease, and an efficiency warning is issued. When e_s>e_y, it is predicted that the working efficiency of the CPython interpreter will not decrease.

[0108] S500, after providing an efficiency warning to the CPython interpreter, calculating a real-time health index of the GIL according to the real-time visual data and GIL feature data;

[0109] The specific steps for calculating the real-time health index of GIL based on real-time visual data and GIL feature data are as follows:

[0110] S501, after issuing an efficiency warning to the CPython interpreter, analyzing the visual data, respectively calculating the difference between the collected real-time visual data and the average value of each visual data before the efficiency decreases in history, and when the difference is greater than 0, determining that the corresponding visual data is status data that affects the decrease in the working efficiency of the CPython interpreter;

[0111] S502, after calculation, the state data that affects the decrease in the working efficiency of the CPython interpreter in the collected real-time visual data is {g1, g2, g3...g j}, g1, g2, g3...g j The first, second, third, ...jth state data that affect the efficiency of the CPython interpreter are determined, where j is a positive integer, 1 <j<m;

[0112] S503, using the GIL feature data set to filter the data that affects the decline in the efficiency of the CPython interpreter, extracting the state data of the GIL feature data in the set in S502 as {w1, w2, w3...w f}, w1, w2, w3...w f is the characteristic data of the 1st, 2nd, 3rd...fth GIL in the data set that affects the decline in the efficiency of the CPython interpreter, where f is a positive integer;

[0113] S504. Calculate the real-time GIL health index using the formula:

[0114]

[0115] In the formula, he_s is the real-time GIL health index, f is the number of GIL feature data among the j status data that affect the decline in CPython interpreter efficiency, j is the total number of status data that affect the decline in CPython interpreter efficiency, sw e is the real-time data value of f GIL feature data in the status data that affects the efficiency of the CPython interpreter. u is the data value of j status data that affects the efficiency of CPython interpreter, 1 <e<f,1<u<j。

[0116] When the CPython interpreter is working and its efficiency drops, it is impossible to simply judge that the GIL is in an unhealthy state based on the efficiency drop, because there are various factors that can cause the working efficiency of the CPython interpreter to drop. Therefore, after an efficiency warning is issued, it is necessary to calculate the health index of the GIL to determine whether the drop in the working efficiency of the CPython interpreter is caused by an unhealthy GIL. If so, a warning is issued; if not, the possibility of the GIL being in an unhealthy state is excluded, and other reasons for the drop in the working efficiency of the CPython interpreter are searched for.

[0117] S600. Determine whether the real-time health index of the GIL is in an unhealthy state according to the GIL health status threshold, and issue a GIL health warning.

[0118] The specific steps for issuing a GIL health warning are as follows:

[0119] Use the GIL health status threshold to judge the GIL health index calculated in real time. When he_s ≥ he_y, judge that the GIL is in an unhealthy state and issue a warning; when he_s < he_y, judge that the GIL is in a healthy state and exclude the possibility that the unhealthy GIL causes the drop in the working efficiency of the CPython interpreter.

[0120] A GIL health status monitoring system based on big data, which includes a data collection module, a data analysis module, a threshold calculation module, a working efficiency warning module, and a GIL status warning module;

[0121] The data collection module is used to collect the working status data of the CPython interpreter and the GIL health status data in history;

[0122] The data analysis module is used to analyze the historical data collected and calculate the visible data set of the drop in the working efficiency of the CPython interpreter and the feature data set of the GIL;

[0123] The threshold calculation module is used to calculate the working efficiency threshold of the CPython interpreter and the GIL health status threshold;

[0124] The working efficiency warning module is used to judge the real-time working efficiency of the CPython interpreter according to the working efficiency threshold and issue a warning;

[0125] The GIL status warning module is used to judge the real-time GIL status according to the GIL health status threshold and issue a warning.

[0126] The data analysis module includes a visible data calculation unit and a GIL feature data calculation unit;

[0127] The visual data calculation unit is used to calculate the state data before and after the work efficiency of the CPython interpreter in history decreases, obtain the difference before and after the work efficiency decreases for each state data, judge the difference of each state data, and obtain the visual data set of the work efficiency decrease;

[0128] The GIL characteristic data calculation unit is used to collect and judge the status data of the CPython interpreter when the GIL is healthy and unhealthy in history, and analyze the status data of the GIL in different states and the same state respectively to obtain the characteristic data set of the GIL.

[0129] The threshold calculation module includes a work efficiency threshold calculation unit and a GIL health status threshold calculation unit;

[0130] The work efficiency threshold calculation unit is used to perform difference calculation on the visual data in the work of the CPython interpreter in history, and calculate the work efficiency threshold of the CPython interpreter by the average value and number of the difference of the visual data in history;

[0131] The GIL health status threshold calculation unit is used to calculate the proportion of the number of GILs in an unhealthy state in the total number of changed data when the working efficiency of the CPython interpreter decreases in history, and obtain the GIL health status threshold.

[0132] Example 1

[0133] Now we collect and analyze the historical working data of CPython interpreter, and get the visible data of the decline of working efficiency of CPython interpreter as {CPU utilization, memory usage, garbage collection time, response time, execution time, thread concurrency performance};

[0134] After collecting and analyzing the CPython interpreter status data when GIL was in different states in history, the characteristic data of GIL was obtained as {response time, thread concurrency performance};

[0135] The visual data of the CPython interpreter's work efficiency is now monitored in real time. The values ​​of the real-time collected visual data are 70%, 349M, 0.4s, 0.1s, 0.03s, and 10 respectively; the average values ​​of each visual data calculated based on historical data are 80%, 349, 0.3, 0.02s, 0.01, and 20 respectively; the work efficiency index is calculated according to the following formula:

[0136]

[0137] The calculated result is e_s=10.49. Assume that the work efficiency threshold calculated based on historical data is 9.8. It is judged that the work efficiency of the CPython interpreter may decrease, and an efficiency warning is issued.

[0138] Then, the visible data collected in real time are calculated and changed, including CPU utilization, garbage collection time, response time, execution time, and thread concurrency performance; the characteristic data are response time and thread concurrency performance; the GIL health index is calculated using the formula:

[0139]

[0140] The calculated he_s=38%, and the calculated GIL health threshold is 70%. It is judged that the GIL is in a healthy state, and the decrease in the working efficiency of the CPython interpreter caused by the GIL is excluded.

[0141] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device.

[0142] Finally, it should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art can still modify the technical solutions described in the aforementioned embodiments or replace some of the technical features therein by equivalents. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A GIL health status monitoring method based on big data, characterized by: The method comprises the following steps: S100, collecting historical status data before and after the work efficiency of the CPython interpreter decreases, performing comparative analysis to classify the status data, and searching and generating a visual data set of the work efficiency decrease of the CPython interpreter; S200, collecting historical status data of the CPython interpreter when the GIL is in an unhealthy state, analyzing and classifying the status data, and searching and generating a feature data set of the GIL; S300, analyzing the historical data of the CPython interpreter when it was working, and calculating the CPython interpreter working efficiency threshold and the GIL health status threshold respectively; The specific steps to calculate the CPython interpreter efficiency threshold and the GIL health status threshold are: S301, calculate the average value s_p of each visual data before the efficiency decreases in the history of the CPython interpreter, collect the data difference of each state data in the visual data set before and after the h times of the CPython interpreter's work efficiency decreases in the history, and calculate the average value of each visual data difference, the formula is: In the formula, C_p is the average value of the difference before and after the work efficiency of each visual data decreases, c_L is the difference of each visual data before and after the work efficiency decreases collected h times, and h is the number of the difference of each visual data; finally, the average value of the difference before and after the work efficiency decreases of m kinds of visual data is calculated; S302, collecting the average number a of changes in visible data when the working efficiency of the CPython interpreter decreases in history, and calculating the working efficiency threshold of the CPython interpreter, the formula is: In the formula, e_y represents the efficiency threshold of the CPython interpreter, a represents the average number of changes in the visual data when the efficiency of the CPython interpreter decreases in history, C_p represents the average difference between each visual data before and after the efficiency decreases, and m is the total number of visual data; S303, collect the total number of status data that changes after the CPython interpreter's working efficiency decreases when the GIL is in an unhealthy state, and calculate the GIL health state threshold. The formula is: In the formula, he is the GIL health status threshold, and b is the number of GIL feature data in the status data that changes after the CPython interpreter's work efficiency decreases; S400, collecting a visual data set in real time when the CPython interpreter is working, and performing real-time monitoring and early warning on the CPython interpreter according to a work efficiency threshold; S500, after providing an efficiency warning to the CPython interpreter, calculating a real-time health index of the GIL according to the real-time visual data and GIL feature data; The specific steps for calculating the real-time health index of GIL based on real-time visual data and GIL feature data are as follows: S501. After an efficiency warning is issued for the CPython interpreter, analyze the visual data, calculate the difference between the average values of each type of visual data collected in real time and the average values of each type of visual data before the efficiency decline in history respectively. When the difference is greater than 0, determine that the corresponding visual data is the status data that affects the decline in the working efficiency of the CPython interpreter; S502, after calculation, the state data that affects the decrease in the working efficiency of the CPython interpreter in the collected real-time visual data is {g1, g2, g3...g j }, g1, g2, g3...g j The first, second, third, ...jth state data that affect the efficiency of the CPython interpreter are determined, where j is a positive integer, 1 <j<m; S503, using the GIL feature data set to filter the data that affects the efficiency of the CPython interpreter, extracting the state data of the GIL feature data in the S502 set as {w1, w2, w3...w f }, w1, w2, w3...w f is the characteristic data of the 1st, 2nd, 3rd...fth GIL in the data set that affects the decline in the efficiency of the CPython interpreter, where f is a positive integer; S504. Calculate the real-time GIL health index, and the formula is: In the formula, he_s is the real-time GIL health index, f is the number of GIL feature data among the j status data that affect the decline in CPython interpreter efficiency, j is the total number of status data that affect the decline in CPython interpreter efficiency, sw e is the real-time data value of f GIL feature data in the status data that affects the efficiency of the CPython interpreter. u is the data value of j status data that affects the efficiency of CPython interpreter, 1 <e<f,1<u<j; S600. Determine whether the real-time health index of GIL is in an unhealthy state according to the GIL health status threshold, and issue a GIL health warning.

2. The GIL health status monitoring method based on big data according to claim 1 is characterized in that: The specific steps for finding and generating the visual data set of the CPython interpreter's working efficiency decline in S100 are as follows: S101. Collect the E-time working time and workload of the CPython interpreter in history, and calculate the working efficiency of the CPython interpreter. The formula is: In the formula, G is the total amount of work, D is the working time, and then the average of the work efficiency of E times is calculated. It is judged that when the work efficiency is lower than the average value of α, the work efficiency decreases; S102, collecting the status data of the CPython interpreter before and after the work efficiency decreases in history, and classifying the status data at the same time as the same type of status data according to the working time period of the CPython interpreter, assuming that the collected status data of the CPython interpreter before and after the work efficiency decreases are {q1, q2, q3...q n }, {q'1, q'2, q'3...q' n }, q1, q2, q3...q n The 1st, 2nd, 3rd, ... nth state data before the efficiency of the CPython interpreter decreases, q ’ 1.q ’ 2.q ’ 3...q ’ n It is the 1st, 2nd, 3rd...nth state data after the efficiency of CPython interpreter decreases, where n is a positive integer; S103. Compare n types of status data before and after the decline in the working efficiency of the CPython interpreter, and calculate the difference of each type of status data before and after the decline in working efficiency respectively. The formula is: C=|s i -s′ i | In the above formula, C is the difference between the state data before and after the work efficiency decreases, s i is the state data value before the work efficiency decreases, s’ i is the state data value after the work efficiency decreases, 0 < i < n, and i is a positive integer; S104, after calculating the difference between each state data before and after the work efficiency decreases, n kinds of difference data {C1, C2, C3...C n }, C1, C2, C3...C n is the calculated difference of the 1st, 2nd, 3rd...nth state data, n is a positive integer, and the standard deviation of the difference of n state data is calculated as follows: In the above formula, C_b is the standard deviation of the differences of n types of status data, C_p is the average value of the differences of n types of status data, and 0 < i < n; S105, judging the visual data that affects the working efficiency of the CPython interpreter according to the calculated standard deviation of the difference values ​​of the n kinds of state data, specifically: when C i >C_b, judge C i The corresponding status data is the visual data L that affects the efficiency of the CPython interpreter. i ≤C_b, judge C i The corresponding state data is not visible data that affects the working efficiency of the CPython interpreter, so it is discarded; after judging n kinds of state data in turn, the visible data set that affects the working efficiency of the CPython interpreter is {L1, L2, L3...L m }, L1, L2, L3...L m The first, second, third, ...m types of visual data that affect the efficiency of the CPython interpreter are determined, where m is a positive integer, 1 <m<n。 3. The GIL health status monitoring method based on big data according to claim 1, characterized in that: The specific steps for finding and generating the feature data set of GIL in S200 are as follows: S201, respectively collect the status data of the CPython interpreter when the GIL is in an unhealthy state and a healthy state in history, and classify the status data at the same time as the same status data according to the working time period of the CPython interpreter. Suppose the status data of the CPython interpreter when the GIL is in a healthy state and an unhealthy state are {p1, p2, p3...p n }, {p'1, p'2, p'3...p' n }, p1, p2, p3...p n The collected data are the 1st, 2nd, 3rd, ... nth states of the CPython interpreter when the GIL is in a healthy state, p'1, ​​p'2, p'3, ... p' n The collected CPython interpreter status data of 1st, 2nd, 3rd...nth when GIL is in an unhealthy state, where n is a positive integer; S202. Compare the collected status data p of the CPython interpreter when the GIL is in a healthy state and an unhealthy state. i and p' i ,After comparison, the state data types that change when GIL is in different states are used as the ,associated data sets of GIL; S203. After obtaining the associated data of GIL by comparison, compare the same type of status data when GIL is in a healthy state and the same type of status data when GIL is in an unhealthy state, extract the data types in which the status data in the CPython interpreter's work changes while the status of GIL remains unchanged, and organize the extracted data types to obtain the GIL disruption data set; S204. Compare the associated data set of GIL with the disruption data set,剔除 the disruption data in the associated data set, and use the data set after剔除 as the feature data set of GIL.

4. The GIL health status monitoring method based on big data according to claim 1, characterized in that: The specific steps for real-time monitoring and warning of the CPython interpreter according to the working efficiency threshold in S400 are as follows: S401. Collect the visual data of the CPython interpreter in real time during work, and calculate the real-time efficiency index according to the collected visual data. The formula is: In the formula, e_s is the efficiency index of the CPython interpreter at work, m is the number of visible data, and ss is r The real-time data value of each visual data, s_p is the average value of each visual data before the efficiency decreases in history, 1 <r<m; S402. When e_s ≥ e_y, predict that the working efficiency of the CPython interpreter will decline and issue an efficiency warning. When e_s > e_y, predict that the working efficiency of the CPython interpreter will not decline.

5. The GIL health status monitoring method based on big data according to claim 1 is characterized by: The specific steps for issuing a GIL health warning in S600 are as follows: Use the GIL health status threshold to judge the real-time calculated GIL health index. When he_s ≥ he_y, judge that GIL is in an unhealthy state and issue a warning; when he_s < he_y, judge that GIL is in a healthy state and rule out that the decline in the working efficiency of the CPython interpreter is caused by the unhealthy GIL.

6. A GIL health status monitoring system based on big data using a GIL health status monitoring method based on big data as claimed in any one of claims 1 to 5, characterized in that: The GIL health status monitoring system includes a data collection module, a data analysis module, a threshold calculation module, a working efficiency warning module, and a GIL status warning module; The data collection module is used to collect the working status data of the CPython interpreter and the GIL health status data in history; The data analysis module is used to analyze the collected historical data and calculate the visual data set of the decline in the working efficiency of the CPython interpreter and the characteristic data set of the GIL; The threshold calculation module is used to calculate the CPython interpreter work efficiency threshold and the GIL health status threshold; The work efficiency warning module is used to judge the work efficiency of the real-time CPython interpreter according to the work efficiency threshold and issue a warning; The GIL status warning module is used to judge the real-time GIL status according to the GIL health status threshold and issue a warning.

7. The GIL health status monitoring system based on big data according to claim 6 is characterized by: The data analysis module includes a visual data calculation unit and a GIL feature data calculation unit; The visual data calculation unit is used to calculate the state data before and after the work efficiency of the CPython interpreter in history decreases, obtain the difference before and after the work efficiency decreases for each state data, judge the difference of each state data, and obtain the visual data set of the work efficiency decrease; The GIL characteristic data calculation unit is used to collect and judge the status data of the CPython interpreter when the GIL is healthy and unhealthy in history, and analyze the status data of the GIL in different states and the same state respectively to obtain the characteristic data set of the GIL.

8. The GIL health status monitoring system based on big data according to claim 6 is characterized by: The threshold calculation module includes a work efficiency threshold calculation unit and a GIL health status threshold calculation unit; The work efficiency threshold calculation unit is used to perform difference calculation on the visual data in the work of the CPython interpreter in history, and calculate the work efficiency threshold of the CPython interpreter by the average value and number of the difference of the visual data in history; The GIL health status threshold calculation unit is used to calculate the proportion of the number of GILs in an unhealthy state in the total number of changed data when the working efficiency of the CPython interpreter decreases in history, and obtain the GIL health status threshold.

Images