An edge computing method for mass data analysis

Abstract

The present application relates to the technical field of data processing, in particular to an edge computing method for massive data analysis. When processing massive Web server access logs, the present application combines log field structure, field complexity and data magnitude to clearly define resource configuration requirements, accurately estimates thread, memory and IO calls, further constructs a behavior sequence based on response time, completion rate and interruption in server behavior characteristics, forms a dynamic evaluation standard for scheduling capability, integrates the grey correlation scoring method between resource requirements and behavior capability in candidate node screening, optimizes the sorting in combination with real-time resource occupation state of the node, makes the task allocation path have better balance in the perspective of multi-dimensional resource utilization, realizes parallel processing through log segment mapping after task scheduling, and forms structured analysis output through edge side summarization, thereby improving the adaptability of edge node scheduling and the accuracy of task allocation.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, and in particular to an edge computing method for massive data analysis. BACKGROUND

[0002] The technical field of data processing involves the processes of collecting, transforming, storing and analyzing digital information, aiming to extract useful information from raw data to support intelligent decision-making and system optimization.

[0003] Among them, the traditional edge computing method for massive data analysis refers to the preliminary data filtering, compression and analysis operation on the edge node closer to the data source, when the data generation is too large, the processing center node processing capacity is insufficient or the bandwidth bottleneck limits the overall performance problem, in order to reduce the transmission load and response delay.

[0004] The traditional edge computing method mainly relies on the geographical location advantage of the node close to the data source to perform simple preliminary filtering and compression, although it reduces the load of the center node, but when the data flow increases sharply or the node running state fluctuates, it lacks the dynamic matching ability between task resource demand and node actual capacity, which is easy to cause uneven node distribution or scheduling conflict, without combining the complexity of log field structure and resource call characteristics, the resource evaluation is rough, which cannot accurately judge the load level required for task execution, for example, when the log field type is highly diverse but the node configuration is fixed, the task execution time may be seriously overestimated, in addition, the node selection only depends on the static performance or simplified index sorting, which cannot make efficient scheduling strategy in the resource occupation tense scene, resulting in part of the node being interrupted or performance bottleneck and affecting the overall task processing efficiency. SUMMARY

[0005] The purpose of the present application is to solve the shortcomings in the prior art and propose an edge computing method for massive data analysis.

[0006] In order to achieve the above purpose, the technical scheme adopted by the present application is as follows: an edge computing method for massive data analysis, comprising the following steps:

[0007] S1: obtaining the field number, log entry number and field type number of the Web server access log, and constructing a resource configuration list of the corresponding log data analysis task;

[0008] S2: the node running data reported by the edge computing server, the behavior index in the node running data is weighted, and the behavior time sequence feature is constructed;

[0009] S3: filtering a scheduling candidate node set by calculating a grey correlation degree score between a behavior time sequence feature of the edge computing server and the resource configuration list of the log data analysis task;

[0010] S4: analyzing resource occupation of each node in the scheduling candidate node set under resource demand of the log data analysis task, sorting all nodes to obtain an adjusted scheduling path set;

[0011] S5: dispatching the web server access log analysis task to each edge computing server according to the adjusted scheduling path set to obtain a data edge computing result.

[0012] As a further scheme of the present application, the resource configuration list comprises thread demand, memory call quantity and channel concurrency value, the behavior time sequence feature comprises response time delay trend, completion rate sequence and interruption number change quantity, the scheduling candidate node set comprises node identification, correlation degree score value and screening bit sequence, the scheduling path set comprises path number, node sorting position and resource occupation comparison value, and the data edge computing result comprises log analysis output item, node processing result and task execution mark.

[0013] As a further scheme of the present application, the obtaining step of the resource configuration list is specifically:

[0014] S111: obtaining three data items of field quantity, log entry quantity and field type quantity in the web server access log, judging field structure composition according to the field quantity and the field type quantity, combining record total quantity information represented by the log entry quantity to establish a field structure record;

[0015] S112: identifying a proportion of the field type in all the field structure records, combining field quantity and type distribution to constitute a field occupation structure, and calculating log structure complexity in a field dimension;

[0016] S113: combining the log structure complexity and data magnitude reflected by the field structure record to induce requirements of CPU thread call quantity, memory usage and IO scheduling frequency of a corresponding log task in a processing process and to normalize processing, and obtaining a resource configuration list.

[0017] As a further scheme of the present application, the obtaining step of the behavior time sequence feature is specifically:

[0018] S211: obtaining node running data reported in continuous multiple scheduling periods of the edge computing server, extracting three behavior indexes of task response time delay, processing completion rate and interruption rollback number, and constructing a period behavior fluctuation sequence according to distribution of the indexes in each period;

[0019] S212: Based on the periodic behavior fluctuation sequence, the periodic increase and decrease change of the task response time delay is compared with the continuous periodic average value of the processing completion rate, the variation range of the number of interrupt rollback times in the period is combined, the trend difference of the three behavior indicators in the time dimension is recognized, and the index variation trend characteristics are obtained;

[0020] S213: According to the index variation trend characteristics, the processing completion rate and the number of interrupt rollback times are constructed into a relative relationship, a multi-dimensional feature set is formed in combination with the task response time delay, and a behavior time sequence feature is generated.

[0021] As a further scheme of the application, the obtaining step of the scheduling candidate node set is specifically:

[0022] S311: The behavior time sequence feature of the edge computing server and the resource configuration list of the log data analysis task are called, and the behavior-resource association relationship group is constructed according to the corresponding relationship of the parameter in the numerical distribution, the change direction and the relative position;

[0023] S312: The gray correlation degree score between each index item in the behavior time sequence feature and each resource item in the resource configuration list is calculated by applying the gray correlation analysis algorithm, the scores corresponding to all edge computing servers are extracted, and a gray correlation degree score result is obtained;

[0024] S313: According to the gray correlation degree score result, the median value in the distribution interval of all scores is set as a screening standard, server nodes with scores higher than the screening standard are selected, and a scheduling candidate node set is obtained.

[0025] As a further scheme of the application, the obtaining step of the adjusted scheduling path set is specifically:

[0026] S411: The running state of the scheduling candidate node set under the resource demand of the log data analysis task is obtained, the CPU usage, memory occupancy and IO channel occupancy rate of each edge computing server under the current task condition are extracted, and a node resource usage condition group is constructed;

[0027] S412: Each node resource usage feature in the node resource usage condition group is compared and analyzed with the demand parameters in the task resource configuration list, and an evaluation and sorting is performed through the analytic hierarchy process, and a node resource sorting result is obtained;

[0028] S413: According to the node resource sorting result, a specified server node path is extracted, and an adjusted scheduling path set is obtained.

[0029] As a further scheme of the application, the obtaining step of the data edge computing result is specifically:

[0030] S511: According to the adjusted scheduling path set, the web server access log analysis task is dispatched to each edge computing server in path order, and a log segment scheduling mapping result is constructed;

[0031] S512: Based on the log segment scheduling mapping result, each edge computing server performs processing operations on the received data segments, extracts log elements in combination with the field information structure, classifies and processes access behaviors, response states and request contents, and generates a log segment local processing result;

[0032] S513: According to the log segment local processing result, the scheduling center integrates and merges the data returned by all edge computing servers, and uniformly constructs the data edge computing result after the completion of the log data analysis task on the edge side.

[0033] Compared with the prior art, the advantages and positive effects of the present application are:

[0034] In the present application, when processing massive web server access logs, the resource configuration requirements are determined by introducing the combination of log field structure, field complexity and data magnitude, the precise estimation of threads, memory and IO calls is realized, the behavior sequence is constructed based on the response time, completion rate and interruption in the server behavior characteristics, the dynamic evaluation standard for scheduling capability is formed, the gray correlation score method between resource requirements and behavior capability is integrated in the candidate node screening, the matching of node selection is improved, the real-time resource occupation state of the node is combined for sorting optimization, the task allocation path has better balance in the aspect of multi-dimensional resource utilization, the parallel processing is realized through log segment mapping after task scheduling, and the structured analysis output is formed through edge side summarization, the adaptability of edge node scheduling and the accuracy of task allocation are improved as a whole, the resource waste and processing delay caused by scheduling imbalance are reduced, and the response speed, resource use efficiency and data processing integrity are significantly enhanced in log analysis execution. BRIEF DESCRIPTION OF DRAWINGS

[0035] Figure 1 The present application is a main step schematic diagram;

[0036] Figure 2 The present application is a flowchart of step S1;

[0037] Figure 3 The present application is a flowchart of step S2;

[0038] Figure 4 The present application is a flowchart of step S3;

[0039] Figure 5 The present application is a flowchart of step S4;

[0040] Figure 6 Flow chart for step S5 of the present application. DETAILED DESCRIPTION

[0041] In order to make the objectives, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific examples described herein are only used to explain the present application and should not be used to limit the present application.

[0042] In the description of the present application, it should be understood that the terms "length", "width", "upper", "lower", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer" and the like indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, and are only used to facilitate the description of the present application and simplify the description, and therefore cannot be understood as indicating or implying that the device or element referred to must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as limiting the present application. In addition, in the description of the present application, the meaning of "a plurality of" is two or more, unless otherwise explicitly and specifically limited.

[0043] Please refer to Figure 1 The present application provides a technical solution: an edge computing method for massive data analysis, comprising the following steps:

[0044] S1: obtaining the number of fields, the number of log entries and the number of field types of the Web server access log, and constructing a resource configuration list corresponding to the log data analysis task;

[0045] S2: reporting node running data on the edge computing server, weighting the behavior indicators in the node running data, and constructing behavior time series features;

[0046] S3: calculating the gray correlation degree score between the behavior time series features of the edge computing server and the resource configuration list of the log data analysis task, and screening a candidate node set;

[0047] S4: analyzing the resource occupation of each node in the candidate node set under the resource demand of the log data analysis task, sorting all nodes, and obtaining an adjusted scheduling path set;

[0048] S5: distributing the Web server access log analysis task to each edge computing server according to the adjusted scheduling path set, and obtaining a data edge computing result;

[0049] The resource configuration list includes thread demand, memory call quantity, and channel concurrency value, the behavior time sequence feature includes response delay trend, completion rate sequence, and interruption number change quantity, the scheduling candidate node set includes node identification, correlation degree score value, and screening bit sequence, the scheduling path set includes path number, node sorting position, and resource occupation comparison value, and the data edge computing result includes log analysis output item, node processing result, and task execution mark.

[0050] Referring to Figure 2 The resource configuration list is obtained by:

[0051] S111: Obtain three data items of field quantity, log entry quantity, and field type quantity in the Web server access log, judge field structure composition according to the field quantity and the field type quantity, establish a field structure record in combination with record total amount information represented by the log entry quantity;

[0052] The three data items of field quantity, log entry quantity, and field type quantity in the Web server access log are obtained. First, each log record is extracted from the log original text and standardized pretreated, all log entries are segmented in a line unit, each line is taken as a log record instance, and a separator such as space, tab or comma is used to split each record to identify the field quantity, for example, a log line is “192.168.1.1--[10 / Mar / 2025:13:55:36+0000]'GET / index.html HTTP / 1.1'2001024”, the field quantity is 10 by using space separation, the total range of the field quantity can be calculated by accumulating all log entries, and a pattern matching technology (such as a regular expression) is used to identify the type of each field, such as an IP address field is “xxx.xxx.xxx.xxx”, a timestamp field is “[xx / Mon / yyyy:hh:mm:ss+zzzz]”, an HTTP method field is “GET”, “POST” and the like, the field type quantity is the total number of different types of fields, if 7 types of fields such as IP address, timestamp, method, URL, protocol, status code, and response size are identified, the field type quantity is 7, the log entry quantity is the number of segmented lines, for example, if a file has 100,000 lines, it is 100,000 records, and the field structure record is constructed based on the above three 4E2A data.

[0053] S112: Identify the proportion of the field type in all field structure records, combine the field quantity and the type distribution to form a field occupation structure, and calculate the log structure complexity according to the field dimension;

[0054] The proportion of the field type in all field structure records is identified, and the field type count of each field structure record is extracted first. The field set is 11 items, including 5 numerical fields, 3 string fields, 2 time fields, and 1 Boolean field. The proportion of each type is as follows: the proportion of numerical fields is the proportion of string fields is the proportion of time fields is the proportion of Boolean fields is In this statistical process, each log record needs to be checked field by field for its value format. For example, if the field “status” is 200, it is an integer field; if “user_agent” is “Mozilla / 5.0”, it is a string field; if “request_time” is “0.215”, it is a floating-point number; if “timestamp” is “2025-06-01T12:30:15”, it is a time field; and if “is_cache_hit” is true or false, it is a Boolean field. The number of the same type of fields in all field structure records is added up to obtain the field type proportion of the whole sample, which constitutes the field occupation structure. Then, the field type proportion is used as the distribution probability to calculate the log structure complexity, which can be expressed by the information entropy formula as follows:

[0055]

[0056] where H is the field structure complexity (unit: bit), representing the uncertainty degree of the field type distribution; n is the total number of field types, for example, there are usually 4 types (integer, string, time, and Boolean); pi is the proportion of the i-th field type in all fields (proportion value), for example, p1=0.45 represents that the first type of field accounts for 45%; and log2 is the logarithm function with base 2, which is used to measure the information entropy unit in bits. i

[0057] For example, the proportion of the field type is as follows: the proportion of numerical fields p1=0.45, the proportion of string fields p2=0.27, the proportion of time fields p3

[0058] =0.18, and the proportion of Boolean fields p4=0.09. Substituting the above formula gives:

[0059] H=-

[0060] (0.45·log20.45+0.27·log20.27+0.18·log20.18+0.09·log20.09)

[0061] ≈1.75;

[0062] ​Further, if another log source field proportion is: numerical p1=0.8, string p2=0.1, time p3=0.1, the structure complexity calculation is:

[0063] H=-(0.8*log2 0.8+0.1*log2 0.1+0.1*log2 0.1)≈0.92;

[0064] It indicates that the field concentration is high, the structure complexity is low, and the field use is concentrated in a few types. If the structure complexity interval is set as follows: -H<1.0: low complexity (single field type, such as most of them are integer or time type); -1.0≤H<1.5: medium complexity (fields have certain diversity); -H≥1.5: high complexity (field types are uniformly distributed and the structure is diverse);

[0065] For example, the server A log contains 12 fields, distributed as: 5 integers, 3 strings, 2 floating-point types, and 2 Boolean types, then Substituting the entropy formula gives:

[0066] H=-

[0067] (0.417*log2 0.417+0.25*log2 0.25+0.167*log2 0.167+0.167*log2 0.167)

[0068] ≈1.89.

[0069] It belongs to high complexity structure. The execution process needs to combine the log field extraction module, field type determination module, field proportion calculation module, structure complexity calculation module, and gradually complete the analysis and complexity output of the field structure.

[0070] S113: Combine the log structure complexity with the data magnitude reflected by the field structure record, summarize the requirements of CPU thread call quantity, memory usage quantity, and IO scheduling frequency of the corresponding log task in the processing process, and normalize the processing to obtain a resource configuration list;

[0071] In combination with the log structure complexity and the data order reflected by the field structure record, firstly, the structure complexity value and the log record entry number of each log need to be obtained, for example, the server A log structure complexity is 1.92, and the total number of records is 180000, the complexity of server B is 0.86, and the record is 32000, by combining the structure complexity and the record number into a data feature group, for example, the feature group corresponding to A is (1.92, 180000), B is (0.86, 32000), and so on, a reference data set is constructed by summarizing multiple server samples, then the CPU thread call quantity, memory usage and IO scheduling frequency value corresponding to each group of features in the sample data need to be analyzed, if the resource occupation of server A corresponds to thread number 8, memory occupation 3.8GB, IO scheduling 150 times per second, server B corresponds to thread number 3, memory occupation 1.2GB, IO scheduling 60 times per second, the dimensions are normalized horizontally for subsequent list standardization, for example, the thread number range is set to a maximum of 16 and a minimum of 2, then the server A thread normalization value is (8-2) / (16-2) = 0.43, and the server B is (3-2) / (16-2) = 0.07, the memory occupation is set to a maximum of 8GB and a minimum of 0.5GB, A is (3.8-0.5) / (8-0.5) = 0.47, B is (1.2-0.5) / (8-0.5) = 0.09, the IO scheduling frequency is maximum 200 times / second and minimum 20 times / second, then A is normalized to (150-20) / (200-20) = 0.72, and B is (60-20) / (200-20) = 0.22, the finally constructed resource configuration list is A: thread 0.43, memory 0.47, IO frequency 0.72, B: thread 0.07, memory 0.09, IO frequency 0.22, the normalized resource indicators form a three-dimensional vector, representing the demand intensity of different log structures for system resources when executing tasks, if used for task scheduling, the normalization result can be converted into a weight basis for resource allocation sorting, for example, when the remaining memory of the resource pool is 2GB, the thread resource is 6, and the disk supports IO frequency 100 times / second, the task with lower comprehensive resource demand index is preferentially allocated, such as B task resource demand is less, and the proportion is low, which can be preferentially executed, and the normalization value of multiple samples is further used to construct a resource threshold reference, for example, the thread normalization value exceeding 0.75 is considered as high concurrency demand, and the value below 0.25 is considered as low thread dependent task, the memory normalization value exceeding 0.65 is determined as high memory occupation, and the IO frequency normalization value higher than 0.7 is considered as intensive IO operation task, the association mapping process between log structure and resource demand is completed in this way, and finally the standardized resource configuration data table is output.

[0072] Please refer to Figure 3 , the behavior time sequence feature acquisition step is specifically:

[0073] S211: Obtain the node running data reported by the edge computing server in multiple consecutive scheduling periods, extract three behavior indicators of task response time delay, processing completion rate and interruption rollback times, and construct a period behavior fluctuation sequence according to the distribution of the indicators in each period;

[0074] To obtain the node running data reported by the edge computing server in multiple consecutive scheduling periods, first, the running log or status report data in each scheduling period is collected in chronological order. In each period data, three key behavior indicators are extracted, namely task response time delay, processing completion rate and interruption rollback times. The task response time delay refers to the time consumed from the task initiation to the node feedback result, which is obtained by the time stamp field difference in the scheduling record. For example, the task is initiated at 12:00:01 and the response is completed at 12:00:04, then the task response time delay is 3 seconds. The processing completion rate is calculated by dividing the number of successfully processed tasks in each period by the total number of tasks, for example, 85 tasks are successfully processed in a period and the total number of tasks is 100, then the completion rate is 85%. The interruption rollback times refer to the number of times the node is rescheduled due to abnormal termination of the task, which can be accumulated and summed by the failed retry record field, for example, there are 4 tasks triggering the rollback process in a period, then the interruption rollback times in this period is 4 times. The three indicators in all scheduling periods are recorded and saved in chronological order to form a period behavior data table, then the distribution of each indicator value in the period is arranged to construct a period behavior fluctuation sequence, for example, the three data of task response time delay, processing completion rate and interruption rollback times in 10 consecutive scheduling periods are arranged in order, which can be used for subsequent trend judgment and behavior evaluation. The whole process needs to rely on the consistency of period division and the integrity of node reported data to ensure the comparability of behavior data, and finally a structured period fluctuation sequence data set is obtained.

[0075] S212: Based on the period behavior fluctuation sequence, compare the period-by-period increase and decrease of the task response time delay with the continuous period average of the processing completion rate, and combine the fluctuation range of the interruption rollback times in the period to identify the trend difference of the three behavior indicators in the time dimension, and obtain the indicator variation trend characteristics;

[0076] Based on the periodic behavior fluctuation sequence, the periodic variation trend of the task response time delay is analyzed, and the difference value between the current period and the previous period is calculated. For example, if the response time delay of a certain scheduling period is 4 seconds and the response time delay of the previous period is 3.6 seconds, the increase is 0.4 seconds. If the response time delay of the next period is 4.3 seconds, the increase continues. The increase and decrease variation sequence of the response time delay is constructed. The processing completion rate is smoothed by taking the average value of the consecutive periods to determine the trend direction. For example, if the completion rates of three consecutive periods are 85%, 88%, and 86%, the average value is (85+88+86) / 3=86.33%. By observing the trend of the response time delay and the completion rate, it is identified whether the two indicators increase or decrease simultaneously. The change amplitude of the interrupt rollback number is introduced to analyze its influence. The change of the interrupt rollback number can be recorded by the absolute value of the difference between adjacent periods. For example, if the rollback number of the current period is 5 and the rollback number of the previous period is 3, the change amplitude is 2. By observing whether the change is consistent with the response time delay fluctuation or whether the fluctuation direction is opposite to the completion rate, the mutual trend relationship of the three indicators on the time axis is determined. If the response time delay continues to rise and the completion rate fluctuates downward, and the rollback number increases frequently, it can be considered that the system is not stable under the current resource or load scheduling strategy. The change curve characteristics and mutual difference trend of the three behavior indicators in the time dimension are output.

[0077] S213: According to the index variation trend characteristics, the processing completion rate and the interrupt rollback number are constructed as a relative relationship, and a multi-dimensional feature set is formed by combining the task response time delay to generate a behavior time sequence feature;

[0078] According to the index variation trend characteristics, the processing completion rate and the interrupt rollback number are established as a relative relationship. For example, the completion rate value and the rollback number value of each period are compared in proportion. For example, if the completion rate of a certain period is 90% and the interrupt rollback is 3 times, the success task ratio corresponding to each unit rollback is 90 / 3=30. If the completion rate of another period is 80% and the rollback is 4 times, the ratio is 80 / 4=20, indicating that the rollback cost increases. Through this relative ratio, a stability reaction coefficient can be formed. Combined with the task response time delay data of the corresponding period, a three-dimensional feature set is constructed. The completion rate represents the system efficiency, the rollback number reflects the stability, and the response time delay reflects the processing speed. The three indicators are combined into a feature vector, such as the completion rate of period 1 is 85%, the rollback number is 3 times, and the response time delay is 4 seconds. The values of period 2 are 88%, 2 times, and 3.8 seconds. The three groups of values are concatenated in order to construct a continuous behavior time sequence feature set for subsequent processing of system state modeling or anomaly detection. The entire construction process is iterated in each scheduling period. The ratio of the completion rate to the rollback number is calculated, and the joint feature value sequence of the three periodic indicators is constructed by cooperating with the time delay to realize the time sequence representation of multi-period behavior data.

[0079] Please refer to Figure 4The obtaining step of the scheduling candidate node set is specifically:

[0080] S311: Call the behavior time sequence feature of the edge computing server and the resource configuration list of the log data analysis task, and construct the behavior resource association relationship group according to the corresponding relationship of the parameters in the numerical distribution, change direction and relative position.

[0081] The call behavior time sequence feature and the resource configuration list provided by the log data analysis task need to first ensure the correspondence consistency in the scheduling cycle time dimension. For each server, extract its behavior time sequence feature data set, which includes task response time, processing completion rate and interruption rollback number in multiple scheduling cycles, and extract CPU thread call amount, memory usage and IO scheduling frequency of the server in the same cycle from the resource configuration list. Next, the corresponding relationship of each parameter in three dimensions needs to be analyzed: first, the numerical distribution dimension, for example, whether the response time and the CPU thread usage are in the same high range, such as the response time appearing more than 5 seconds and the thread call number more than 10, which constitutes the same numerical distribution, second, the change direction dimension, that is, whether the two data increase or decrease at the same time in the cycle, for example, in cycle 1 to cycle 3, the task response time increases from 3.5 seconds to 4.2 seconds, and then to 5.1 seconds, while the CPU thread call amount increases from 6 to 8, and then to 10, which indicates that the change direction of the two is consistent, and third, the relative position dimension, that is, all servers in the cycle are sorted according to the index, for example, the task response time is ranked 5th among 20 servers, and the thread call amount is ranked 4th, which indicates that the two have similar relative positions in the current cycle. The corresponding conditions of the above three dimensions are collected, and each index item in the behavior feature of each server is paired with each resource item in the resource configuration list to form 9 groups of behavior and resource corresponding items. Through the above analysis, the correlation degree of each paired item in the numerical, change direction and position dimension is judged, and the results of all paired items are collected in order according to the server number, and finally the behavior resource association relationship group is constructed for each edge computing server.

[0082] S312: Apply gray correlation analysis algorithm to calculate the gray correlation degree score between each index item in the behavior time sequence feature and each resource item in the resource configuration list, extract the corresponding scores of all edge computing servers, and obtain the gray correlation degree score result.

[0083] Applying the grey correlation analysis algorithm requires comparing the behavior time series characteristics of each edge computing server with each resource item in the resource configuration list two by two to calculate the grey correlation scores between the three behavior indicators (task response time, processing completion rate, and interruption rollback times) and the three resource indicators (CPU thread call quantity, memory usage, and IO scheduling frequency). In actual calculation, first, normalize all behavior and resource data, scale the original data to between 0 and 1 according to the range method, and ensure comparability between different dimensions. For example, the task response time data range of a certain server in 10 scheduling periods is from 3.5 seconds to 5.1 seconds, and the normalized standard value of the response time in the first period is calculated as:

[0084] (3.5-3.5) / (5.1-3.5)=0. The thread call quantity in the resource configuration is 6 to 12, and if it is 6 in the first period, it is normalized to 0. After normalization, for each behavior and resource pair, the absolute difference value of each period is calculated to form a difference sequence, and then the grey correlation coefficient of each period is calculated according to the following formula:

[0085]

[0086] wherein, represents the grey correlation coefficient of the i1th behavior indicator (e.g., task response time) and the kth resource indicator (e.g., CPU thread) in the kth scheduling period; represents the absolute difference value of the i1th behavior indicator and the kth resource indicator after normalization in the period; Δ min : the minimum value of the difference value in all periods and pair combinations; Δ max : the maximum value of the difference value in all periods and pair combinations; ρ: resolution coefficient, commonly taking a value of 0.5, used to adjust the sensitivity of differentiation;

[0087] If the task response time and CPU thread usage two standardized sequences have a difference value of 0.05 in 10 periods, and the maximum difference value and the minimum difference value are both 0.05, then:

[0088]

[0089] represents complete correlation. If the difference value of another pair of combinations is 0.1, 0.25, 0.3, etc., the maximum difference value is 0.4, and the minimum is 0.1, then the correlation coefficient of the first period is:

[0090]

[0091] The second period is:

[0092]

[0093] After all the correlation coefficients of the cycles are completed, their average value is calculated as the final gray correlation degree score of the behavior-resource index pair, for example, the average of 10 cycles is 0.81. This calculation process is performed on the three behavior indicators and the three resource indicators in nine groups of pairing, and the average score is calculated respectively, and finally nine score values corresponding to each edge node are formed as the quantitative representation of the behavior-resource relationship, and the gray correlation degree score extraction process is completed.

[0094] S313: According to the gray correlation degree score result, the median value in the distribution interval of all scores is set as the screening standard, and the server nodes with scores higher than the screening standard are selected to obtain the scheduling candidate node set;

[0095] First, the nine score values of all servers are collected and sorted according to the pairing relationship of behavior and resource items to form a score set for each pairing item, for example, the scores of "response time and CPU threads" in all servers form a list, and the scores of "processing completion rate and memory occupation" form another list, and so on. Then, each score set is sorted and the median value of its distribution interval is calculated to construct the screening standard, for example, if the "response time and CPU thread" score set contains 20 values, the average of the 10th and 11th values in the middle position is taken as the median value screening standard after arranging them from small to large. Then, the scores of all server nodes in this pairing item are compared with the corresponding screening standard, and if the score is higher than the median value, it is considered to have strong behavior correlation in this resource relationship dimension. In this way, the nine pairing items are screened and marked respectively, and then the pairing items marked as "higher than the median value" are combined and counted. If a server has six or more scores higher than the corresponding median standard in the nine scores, it is considered that the server has high overall behavior-resource correlation, and is included in the scheduling candidate node set. Finally, a number of server node number sets are formed as the resource scheduling candidate list with preferred characteristics.

[0096] Please refer to Figure 5 The steps of obtaining the adjusted scheduling path set are as follows:

[0097] S411: Obtain the running state of the scheduling candidate node set under the resource demand of the log data analysis task, extract the CPU usage, memory occupation and IO channel occupation rate of each edge computing server under the current task condition, and construct a node resource usage group.

[0098] To obtain the running state of the scheduling candidate node set under the resource requirement condition of the log data analysis task, first, the resource running indexes of each edge computing server need to be collected in real time before the task is loaded or at the initial stage of the task, including CPU usage, memory occupancy, and IO channel occupancy rate. The CPU usage can be directly read from the system monitoring tool, the memory occupancy is calculated by the ratio of the current used memory to the total memory in the operating system resource table, for example, if the server uses 3.2 GB of memory and the total memory is 8 GB, the occupancy rate is 40%, and the IO channel occupancy rate can be estimated by the active time ratio of the disk or network IO interface, for example, if the IO active time is 6 seconds in 10 seconds, the IO channel occupancy rate is 60%. The above three data are collected from each server in the scheduling candidate set and recorded in the data structure with the server node as the unit, forming a node resource usage status group. The three index values of each node will be used as an important basis for judging the resource availability and task adaptability. The process needs to keep consistent with the index types in the task resource requirement table and keep real-time to facilitate subsequent node adaptation and sorting evaluation. Finally, the standardized collection of node resource usage characteristics is completed.

[0099] S412: Compare each node resource usage characteristic in the node resource usage status group with the requirement parameter in the task resource configuration list, evaluate and sort by the analytic hierarchy process to obtain the node resource sorting result;

[0100] The resource usage characteristics of each node in the node resource usage status group are compared with the resource configuration parameters required by the log data analysis task. The analytic hierarchy process (AHP) is used to evaluate and sort the server nodes to obtain the priority order of resource adaptation. In the log data analysis task, the task often contains decoding, extraction, filtering, structure analysis and storage of a large number of log lines. Such tasks are usually CPU-intensive and have obvious characteristics, with moderate memory requirements and less IO scheduling frequency. Based on this actual running feature, in this example, the relative importance of the three resource indexes is set as follows: CPU usage is the most critical, memory occupancy is the second, and IO channel occupancy rate is the third.

[0101] The resource importance preference is set as follows: the importance of CPU relative to memory is 3, the importance of CPU relative to IO is 5, and the importance of memory relative to IO is 2. The three-level judgment matrix is constructed as follows:

[0102]

[0103] The sum of each column is calculated respectively: the first column: 1+1 / 3+1 / 5≈1.867, the second column: 3+1+0.5=4.5, the third column: 5+2+1=8.

[0104] Divide each element in the matrix by the column sum of the column it is in to form a normalized matrix:

[0105]

[0106] Take the average of each row to get the weight vector: CPU usage weight:

[0107] (0.535+0.667+0.625) / 3≈0.609, memory usage weight:

[0108] (0.179+0.222+0.25) / 3≈0.217, IO channel occupancy weight:

[0109] (0.107+0.111+0.125) / 3≈0.114.

[0110] Express the current resource usage of each node in a normalized form: Node A: CPU usage 70%, memory usage 40%, IO usage 50%, expressed as [0.7, 0.4, 0.5], Node B: CPU usage 50%, memory usage 30%, IO usage 40%, expressed as [0.5, 0.3, 0.4].

[0111] Weight the sum of the node resource eigenvalues with the weight of each resource as a coefficient to form a comprehensive fitness score, calculated as follows:

[0112] Node A score: S A = 0.609x0.7 + 0.217x0.4 + 0.114x0.5 = 0.5701;

[0113] Node B score: S B = 0.609x0.5 + 0.217x0.3 + 0.114x0.4 = 0.4152.

[0114] Conclusion: Node A score 0.5701, Node B score 0.4152, indicating that the resource structure of Node A is more in line with the resource requirements of the log data analysis task, so it has a higher priority in the ranking. In this way, resource fitness calculations can be completed for all candidate server nodes and a ranking result list can be generated.

[0115] S413: According to the node resource ranking result, extract the specified server node path to obtain an adjusted scheduling path set;

[0116] First, according to the resource adaptation score, all candidate server nodes are sorted in descending order of the score, for example, the system sets the number of selectable paths to 3, then the top 3 nodes are selected as the target node path set. If there are nodes with the same score in the sorting process, secondary screening can be performed according to the standby rules, for example, the CPU usage is lower, or the physical distance to the current task data source is closer, to improve the efficiency and response speed of scheduling. The physical path information of each selected node, including node identifier, access address, service port and network channel configuration, needs to be extracted and recorded to form complete path data. Finally, these path information is combined to form a new scheduling path set.

[0117] Referring to Figure 6 , the data edge computing result obtaining step is specifically:

[0118] S511: According to the adjusted scheduling path set, the web server access log analysis task is dispatched to each edge computing server in path order to construct a log segment scheduling mapping result;

[0119] According to the adjusted scheduling path set, the web server access log analysis task is dispatched to each edge computing server in path order, which needs to first divide the original log data set into equal or proportional slices according to resource capacity, for example, the total log data amount is 1.2 million records, the target scheduling path set contains 4 edge nodes, among which node A has the highest resource score and can allocate 35% of the total amount, nodes B and C have similar scores and each allocate 25%, and node D has the lowest resource and allocates 15%. Then, according to the corresponding proportion, about 420,000, 300,000, 300,000 and 180,000 log records are extracted and distributed to each node, while recording the task number and node mapping relationship in the distribution process to construct a log segment scheduling mapping table. The mapping table needs to contain each node number, the received log start and end row number, data distribution timestamp, allocated field structure type and other information. The entire scheduling mapping is executed in the scheduling center, and then the task segments are pushed to the edge nodes through a distributed transmission mechanism, thereby establishing the path dispatch structure between the task segments and the processing nodes.

[0120] S512: Based on the log segment scheduling mapping result, each edge computing server performs processing operations on the received data segments, extracts log elements combined with field information structure, classifies and processes access behavior, response status and request content, and generates a local processing result of the log segment;

[0121] Based on the log segment scheduling mapping result, each edge computing server needs to independently complete the processing operation of the received data segment. First, the corresponding segment log record is loaded, and the field structure format is parsed. For example, each log contains IP address, access time, request path, status code, user agent, etc. The field values are extracted and stored in a local data table. Then, the access behavior field is classified. For example, the module type can be divided according to the keywords contained in the access path, such as interface call for containing " / api / ", user authentication for containing " / login", and status code field for success, redirection, client error, and server error. The request content field can be further parsed for parameter structure, such as extracting the request method GET, path / product, and parameter id=123 from "GET / product?id=123". All results are arranged in chronological order or request source to form a node-level log segment local processing result set. Each processing result includes access behavior category statistics, status code distribution, request type details, and key field summary, and is labeled with node number and task number.

[0122] S513: According to the log segment local processing result, the scheduling center integrates and merges the data returned by all edge computing servers, and uniformly constructs the data edge computing result after the log data analysis task is completed on the edge side.

[0123] According to the log segment local processing result returned by each edge computing server, the scheduling center uniformly collects the output data of all processing nodes, and splices and merges them according to the task number and log segment number sequence. First, the classification statistics items are processed, such as the "interface call" access times of each node, which are 120,000, 100,000, 90,000, and 60,000, respectively, and the total is 370,000 times. Then, the status code distribution is normalized and synthesized, such as the 2XX success status numbers of each node, which are 240,000, 220,000, 180,000, and 100,000, respectively, and the total is 740,000. The request content can be merged by field splicing and field value unique identification field. If different nodes process repeated request paths or IP access sequences, the latest record needs to be retained according to the access timestamp and source order during merging. After integration, the data output by the scheduling center is the log analysis result data generated by each edge server during task execution, which represents the full completion of the task in a distributed environment under the edge computing architecture.

[0124] The above merely describes the preferred embodiments of the present application, and is not intended to limit the present application in other forms. Any skilled person in the art can modify or change the disclosed technical content into equivalent embodiments with equivalent changes, and apply them to other fields. However, any simple modification, equivalent change and modification made to the above embodiments according to the technical essence of the present application, without departing from the technical solution content of the present application, still falls within the protection scope of the present application.

Claims

1. An edge computing method for mass data analysis, characterized in that, The method comprises the following steps: S1: obtaining the field quantity, log entry quantity and field type quantity of a web server access log, and constructing a resource configuration list corresponding to a log data analysis task; S2: reporting node running data on an edge computing server, performing weighted processing on behavior indicators in the node running data, and constructing behavior time sequence features; S3: calculating the gray correlation degree score between the behavior time sequence features of the edge computing server and the resource configuration list of the log data analysis task, and screening a candidate node set; S4: analyzing the resource occupation of each node in the candidate node set under the resource demand of the log data analysis task, sorting all nodes, and obtaining an adjusted scheduling path set; S5: distributing the web server access log analysis task to each edge computing server according to the adjusted scheduling path set, and obtaining a data edge computing result. The resource configuration list is obtained by: S111: obtaining the field quantity, log entry quantity and field type quantity in the web server access log, judging the field structure composition according to the field quantity and field type quantity, combining the record total quantity information represented by the log entry quantity, and establishing a field structure record; S112: identifying the proportion of the field type in all field structure records, combining the field quantity and type distribution to form a field occupation structure, and calculating the log structure complexity in the field dimension; S113: combining the log structure complexity and the data magnitude reflected by the field structure record, inducing the requirements of CPU thread calling quantity, memory usage and IO scheduling frequency in the processing process of the corresponding log task, and normalizing the processing to obtain the resource configuration list; The behavior time sequence features are obtained by: S211: obtaining the node running data reported in a plurality of continuous scheduling periods of the edge computing server, extracting three behavior indicators of task response time delay, processing completion rate and interruption rollback times, constructing a period behavior fluctuation sequence according to the distribution of the indicators in each period; S212: comparing the period-by-period increase and decrease of the task response time delay with the continuous period average of the processing completion rate, combining the fluctuation range of the interruption rollback times in the period, identifying the trend difference of the three behavior indicators in the time dimension, and obtaining the index variation trend feature; S213: constructing the processing completion rate and the interruption rollback times into a relative relationship according to the index variation trend feature, combining the task response time delay to form a multi-dimensional feature set, and generating the behavior time sequence features. 2.The edge computing method for massive data analysis of claim 1, wherein, The resource configuration list includes thread demand, memory calling quantity and channel concurrency value, the behavior time sequence features include response time delay trend, completion rate sequence and interruption times change quantity, the candidate node set includes node identifier, correlation score value and screening bit sequence, the scheduling path set includes path number, node sorting position and resource occupation comparison value, and the data edge computing result includes log analysis output item, node processing result and task execution marker. 3.The edge computing method for massive data analysis of claim 1, wherein, The obtaining step of the scheduling candidate node set is specifically: S311: Call the behavior time sequence feature of the edge computing server and the resource configuration list of the log data analysis task, and construct the behavior-resource association relationship group according to the corresponding relationship of the parameter in the value distribution, the change direction and the relative position; S312: Calculate the gray correlation degree score between each index item in the behavior time sequence feature and each resource item in the resource configuration list by applying the gray correlation analysis algorithm, extract the scores corresponding to all edge computing servers, and obtain the gray correlation degree score result; S313: According to the gray correlation degree score result, set the median value in the distribution interval of all scores as the screening standard, select the server nodes with scores higher than the screening standard, and obtain the scheduling candidate node set.

4. The edge computing method for massive data analysis according to claim 3, characterized in that, The obtaining step of the adjusted scheduling path set is specifically: S411: Obtain the running state of the scheduling candidate node set under the resource demand of the log data analysis task, extract the CPU usage, memory occupation and IO channel occupation rate of each edge computing server under the current task condition, and construct the node resource usage condition group; S412: Compare and analyze each node resource usage feature in the node resource usage condition group with the demand parameters in the task resource configuration list, evaluate and sort by the analytic hierarchy process, and obtain the node resource sorting result; S413: According to the node resource sorting result, extract the specified server node path, and obtain the adjusted scheduling path set.

5. The edge computing method for mass data analysis according to claim 4, wherein, The obtaining step of the data edge computing result is specifically: S511: According to the adjusted scheduling path set, dispatch the Web server access log analysis task to each edge computing server in path order, and construct the log segment scheduling mapping result; S512: Based on the log segment scheduling mapping result, each edge computing server performs processing operation on the received data segment, extracts log elements combined with field information structure, classifies and processes access behavior, response state and request content, and generates log segment local processing result; S513: According to the log segment local processing result, the scheduling center integrates and merges the data returned by all edge computing servers, and uniformly constructs the data edge computing result after the log data analysis task is completed on the edge side.

Images