Inspection log collection method, device, equipment and storage medium

Abstract

The present application discloses a method, apparatus, device and storage medium for collecting inspection logs. The method comprises: obtaining inspection log source data of at least one device from a cloud inspection platform based on a batch-stream integration mechanism; judging the priority of the inspection log source data of at least one device; determining the collection order of at least one inspection log source data based on the priority of at least one inspection log source data; and calling a batch-stream integration processing engine to collect at least one inspection log source data based on the collection order to obtain at least one inspection log data.

Description

Technical Field

[0001] The present application relates to the field of big data technology, and is related to, but not limited to, a method, apparatus, device, and storage medium for collecting inspection logs. Background Art

[0002] In order to manage different types of equipment such as servers, routers, switches, firewalls, etc. under each resource pool in the network cloud region, it is necessary to build a network cloud automatic inspection platform to open up the automated command channel. Inspection instructions for each device are issued periodically through the command channel. After the regional front-end execution node executes the inspection instructions, a large number of inspection logs are generated.

[0003] However, the inspection log files contain information about the equipment inspection results. The amount of log data is huge and continues to grow. At the same time, the inspection logs generated are unstructured standard data, which is not easy to quickly analyze and associate aggregated statistics. Summary of the Invention

[0004] In view of this, embodiments of the present application provide a method, apparatus, device, and storage medium for collecting inspection logs.

[0005] In the first aspect, an embodiment of the present application provides a method for collecting inspection logs, the method comprising: based on a batch-stream integrated mechanism, obtaining inspection log source data of at least one device from a cloud inspection platform; judging the priority of the inspection log source data of at least one device; based on the priority of at least one of the inspection log source data, determining the collection order of at least one of the inspection log source data; based on the collection order, calling a batch-stream integrated processing engine to collect at least one of the inspection log source data to obtain at least one inspection log data.

[0006] In the second aspect, an embodiment of the present application provides an inspection log collection device, including: an acquisition module, used to obtain the inspection log source data of at least one device from a cloud inspection platform based on a batch-stream integrated mechanism; a judgment module, used to judge the priority of the inspection log source data of the at least one device; a first determination module, used to determine the collection order of at least one inspection log source data based on the priority of at least one inspection log source data; a collection module, used to call the batch-stream integrated processing engine to collect at least one inspection log source data based on the collection order, and obtain at least one inspection log data.

[0007] In a third aspect, an embodiment of the present application provides an electronic device comprising a memory and a processor, wherein the memory stores a computer program that can be run on the processor, and when the processor executes the program, the steps in the inspection log collection method described in the embodiment of the present application are implemented.

[0008] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps in the inspection log collection method described in the embodiment of the present application are implemented.

[0009] In an embodiment of the present application, compared with directly pulling or receiving data from the source data, when resources are limited and a large amount of inspection log source data is waiting to be collected, the dynamic priority batch and stream integration method can enable higher priority inspection log source data to be quickly collected without affecting the collection of lower priority inspection log source data, thereby improving the efficiency of data collection. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 This is a flow chart of a method for collecting inspection logs according to an embodiment of the present application;

[0011] Figure 2 This is a schematic diagram of inspection items included in different types of equipment according to an embodiment of the present application;

[0012] Figure 3 This is a schematic diagram of a queue-grabbing processing method according to an embodiment of the present application;

[0013] Figure 4 This is a schematic diagram of a method for seizing high-priority inspection log source data resources according to an embodiment of the present application;

[0014] Figure 5 This is a flow chart of a log collection method according to an embodiment of the present application;

[0015] Figure 6 This is a schematic diagram of the interaction between a log correlation analysis engine and a cloud inspection platform according to an embodiment of the present application;

[0016] Figure 7 This is a schematic diagram of inspection item rules for different types of equipment according to an embodiment of the present application;

[0017] Figure 8 This is a flow chart of a log preprocessing method according to an embodiment of the present application;

[0018] Figure 9 It is the combined diagram of array and linked list in H-struct;

[0019] Figure 10 This is the structure diagram of the array and linked list in H-struct;

[0020] Figure 11 This is a flowchart of a method for converting data into a data structure for storage and parallel processing according to an embodiment of the present application;

[0021] Figure 12a A schematic diagram of a transaction processing method for multiple queues in the related art;

[0022] Figure 12b Schematic diagram of a transaction processing method for multiple queues in an embodiment of the present application;

[0023] Figure 13 This is a flow chart of a log analysis method according to an embodiment of the present application;

[0024] Figure 14 This is a schematic diagram of the structure of an inspection log collection device according to an embodiment of the present application;

[0025] Figure 15 A schematic diagram of a hardware entity of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0026] The technical solution of the present application is further described in detail below with reference to the accompanying drawings and embodiments.

[0027] To manage the diverse types of devices, such as servers, routers, switches, and firewalls, within each resource pool within a network cloud region, a network cloud automated inspection platform (also known as a cloud inspection platform) is required to establish an automated command channel. This command channel periodically issues inspection commands to each device. After the regional front-end nodes execute these commands, they generate a large number of inspection logs. Inspection log files contain information about device inspection results, and the volume of log data is substantial and continuously growing. Furthermore, the generated inspection logs are unstructured and standardized, making them difficult to quickly analyze and aggregate. Devices are interconnected, and analyzing logs from a single device type cannot address inter-device dependency monitoring. An efficient, reliable, and comprehensive process is required to rapidly analyze and process inspection logs, unlock their value, support multi-dimensional visualization of inspection results, analyze and warn of inspection anomalies, and provide relevant data metrics for disaster recovery decisions and intelligent operations and maintenance of network cloud resource pool devices.

[0028] In related technologies, a log analysis method based on association analysis and a rule base is provided for analyzing logs generated by various systems and gateway devices. First, a regular expression-based pattern matching method is applied to perform preliminary log data processing. Then, an FP-tree (Frequent Pattern-growth) algorithm is used to mine association rules.

[0029] In related technologies, the log collection process only supports batch data collection. The collection efficiency of directly sending log files and indirectly reading databases is not high, and there is a long delay problem.

[0030] In the related art, pattern matching is only performed through regular expressions during log preprocessing, and the preprocessing process is incomplete.

[0031] In the related art, the FP-tree frequency set algorithm is used to mine association rules in the log analysis phase. When faced with massive data sets, it is very difficult to construct a memory-based FP tree, and the algorithm efficiency is very low.

[0032] Figure 1 This is a flow chart of a method for collecting inspection logs according to an embodiment of the present application. Figure 1 As shown, the method includes:

[0033] Step S102: Based on the batch-stream integration mechanism, obtain the inspection log source data of at least one device from the cloud inspection platform;

[0034] Among them, in order to manage different types of devices such as servers, routers, switches, firewalls, etc. under each resource pool in the cloud region, the cloud inspection platform periodically issues inspection instructions to each device through the command channel. After each device executes the inspection instruction, a large amount of inspection log source data is generated. The inspection log source data can also be called inspection log original data. The inspection log source data includes the content information of the equipment inspection results; the log collection module in the log analysis platform can obtain the inspection log source data of multiple devices from the cloud inspection platform based on the batch and stream integration mechanism.

[0035] In the field of big data processing, batch processing tasks and stream processing tasks are generally considered to be two different tasks. Batch processing is to collect and store multiple data records and process these data together in one operation. Stream processing is to continuously monitor the data source and process the data source in real time when new data events occur. A big data framework is generally designed to handle only one of these tasks. The batch-stream integration mechanism can support both batch and stream processing tasks through a flexible batch-stream integration processing engine.

[0036] The log collection module responsible for collecting inspection logs can synchronize inspection log source data from the database, file server, and message queue of the network cloud inspection platform in a batch and stream integrated manner. The database may include a relational database and a non-relational database.

[0037] Step S104: determining the priority of the inspection log source data of the at least one device;

[0038] Among them, a dynamic priority mechanism can be introduced when collecting inspection logs to define the corresponding priority for the inspection log source data of each device. When resources are limited, the machine performance can be fully utilized, and the dynamic configuration of priority can enable the inspection log source data to be quickly and synchronously processed.

[0039] Step S106: determining a collection order of at least one inspection log source data based on a priority of at least one inspection log source data;

[0040] Among them, the higher the priority, the earlier the collection order can be, that is, the inspection log source data with a high priority can be collected first.

[0041] Step S108: Based on the collection order, call the batch-stream integrated processing engine to collect at least one inspection log source data to obtain at least one inspection log data.

[0042] In an embodiment of the present application, compared with directly pulling or receiving data from the source data, when resources are limited and a large amount of inspection log source data is waiting to be collected, the dynamic priority batch and stream integration method can enable higher priority inspection log source data to be quickly collected without affecting the collection of lower priority inspection log source data, thereby improving the efficiency of data collection.

[0043] In some embodiments, step S104 “determining the priority of the inspection log source data of the at least one device” can be implemented by the following steps S141 and S142:

[0044] Step S141: Determine the similarity between inspection items of at least one device, the device link urgency, and the dynamic adjustment factor; the device link urgency is used to indicate the urgency of collecting inspection log source data of the link where the device is located; the dynamic adjustment factor is used to adjust the priority of the inspection log source data of the device;

[0045] The parameters affecting the priority of the inspection log source data may include the similarity R of the inspection items corresponding to the inspection log source data of at least one device, the device link urgency U, and the dynamic adjustment factor D.

[0046] Step S142: determining the priority of the inspection log source data of the at least one device based on the similarity between the inspection items of the at least one device, the device link urgency, and the dynamic adjustment factor.

[0047] Different devices may have identical or similar inspection items. For example, hosts, storage node servers, and physical servers all require inspections for memory and disk size. Most devices also require inspections for items like power status and network connectivity. However, inspection instructions are issued for each device individually, not by inspection item. This results in the order of the returned inspection log source data not necessarily matching the optimal priority. The similarity of inspection items is a key factor influencing priority; the higher the similarity, the higher the priority. The similarity of these inspection items can be pre-evaluated and calculated based on business experience.

[0048] The evaluation method of inspection items may include: independently decomposing various equipment inspection items into N1, N2...N i , the subscript i can represent different types of equipment, such as Figure 2 As shown, it is assumed that it is decomposed into two independent modules N1 and N2. The inspection items in module N1 include memory size check represented by a, CPU (Central Processing Unit) usage represented by b, power status represented by c, system time check represented by d, etc. The inspection items in module N2 include memory size check represented by a, CPU usage represented by b, system time check represented by d, temperature check represented by e, etc.

[0049] Then the similarity R between the inspection items of device N1 and device N2 can be expressed by the following formula (1):

[0050]

[0051] Among them, the total number of inspection items of N1 and N2 is 5, namely a, b, c, d and e. The same inspection items include a, b and d. Therefore, the similarity between N1 and N2 is 3 / 5.

[0052] The urgency of a device link can be determined based on the data integrity (i.e., link integrity) of the inspection log source data. There is a link hierarchy between servers, routers, switches, and other devices. Sometimes, to more quickly present the inspection log status of a particular device link, it is necessary to prioritize synchronizing the inspection log source data on that link. To increase the priority of manual intervention in link data synchronization, we set one to five levels (the default is one) based on the urgency of different device links. The higher the urgency of the device link, the higher the priority, meaning that manual intervention should be greater.

[0053] The default value of the dynamic adjustment factor of the inspection log source data of different devices is the same, which can be expressed as m. Its value can be dynamically adjusted as the log analysis and aggregation are executed. Adding a dynamic adjustment factor can correct the accuracy of the historical priority of the inspection log source data of the device. The priority calculated by the similarity R of the inspection item and the urgency requirement U of the device link may have errors. After the log analysis aggregates the inspection results, they are transmitted back to the log collection module through the message queue. The log collection module consumes the messages into the cache database. After selecting the priority parameters of the inspection log source data, the priority index value of each inspection log source data can be calculated according to the following formula (2):

[0054]

[0055] Among them, λ1 can represent the device N i The similarity R of the inspection items i The weight of device N is represented by λ2. i The device link urgency U i The weight of device N can be represented by λ3. i Dynamic adjustment factor Di When the priority index value T of the inspection log source data is greater than or equal to the default value, the priority of the inspection log source data can be considered as high priority; when the priority index value T of the inspection log source data is less than the default value, the priority of the inspection log source data can be considered as low priority.

[0056] In an embodiment of the present application, the priority of the inspection log source data of at least one device is determined based on the similarity between the inspection items of at least one device, the urgency of the device link and the dynamic adjustment factor, so that the priority of the inspection log source data can be determined more accurately, and the priority can be dynamically adjusted.

[0057] In some embodiments, the step S141 of "determining the device link urgency of at least one device" includes the following steps S1411 to S1413:

[0058] Step S1411: Determine the data attribute integrity, data volume integrity, and upper and lower hierarchical chain integrity of the inspection log source data of at least one device;

[0059] Step S1412: Determine the link integrity of each device based on the data attribute integrity, data volume integrity, and upper and lower layer chain integrity of each device;

[0060] Among them, the measurement factors of data integrity include data attribute integrity, data volume integrity, and upper and lower level chain integrity. The integrity of a column attribute can be expressed as C j , calculate the sum of attribute integrity of all m columns The total amount of data in a period of time can be expressed as g, and the amount of data that meets the link requirements can be expressed as f. The data integrity can be expressed as The integrity of the upper and lower level chains of any chain t is recorded as avg(t i ) represents t i The average value, sum(t i ) represents t i The total value of , where i indicates whether the upper and lower chains have values

[0061] The integrity of the entire device link can be expressed as: C = avg (C mj +C e +C t ), in order to continuously revise the device link integrity index to achieve a better effect, the historical data C1, C2...C of each device link integrity can be recorded. n , first calculate the average value of each link integrity At the same time, in order to measure the fluctuation of historical integrity data, these data fluctuation values ​​are calculated When a new batch of data is collected, the percentage of deviation between the current data and the average value is calculated Refer to the offset percentage and select two points with smaller fluctuation values ​​in historical data x i , x i+1 Correct device link integrity The corrected link integrity is more stable than before, reducing the calculation parameter error. Finally, the integrity value is mapped and bound to the device urgency requirement. Table 1 shows the mapping relationship between device link urgency and data integrity. The intervals of device link data integrity C are 0% <= C < 20%, 20% <= C < 40%, 40% <= C < 60%, 60% <= C < 80%, and 80% <= C <= 100%, corresponding to levels one to five respectively. The lower the data integrity, the higher the level of device link urgency can be, that is, the greater the intensity of manual intervention should be.

[0062] Table 1

[0063] Device link urgency Link data integrity interval U＝1 0％<＝C<20％ U＝2 20％<＝C<40％ U＝3 40％<＝C<60％ U＝4 60％<＝C<80％ U＝5 80％<＝C<＝100％

[0064] Step S1413: Determine the device link urgency of each device based on the link integrity of each device and the preset mapping relationship.

[0065] Among them, referring to Table 1, devices with link data integrity between 40% and 60% have lower data integrity than devices with link data integrity between 60% and 80%. The corresponding device link urgency level is higher, the intensity of manual intervention should be greater, and the priority to be given is greater.

[0066] In an embodiment of the present application, the link integrity of the corresponding device is determined based on the data attribute integrity, data volume integrity and upper and lower level chain integrity of each of the devices, and the link urgency is determined based on the link integrity, so that greater manual intervention can be performed on data with low data integrity and high link urgency.

[0067] In some embodiments, the priority includes a first priority and a second priority, the first priority being higher than the second priority. In step S108, "calling the batch-stream integrated processing engine to collect at least one of the inspection log source data" may be implemented by following steps S182 to S186:

[0068] Step S182: putting at least one first inspection log source data of the first priority into a first queue, and putting at least one second inspection log source data of the second priority into a second queue; the first queue is a mutex lock or spin lock queue;

[0069] Step S184: When calling the batch-stream integrated processing engine to collect the second target inspection log source data in the second queue, triggering at least one first inspection log source data in the first queue to perform resource preemption processing, so that the first target inspection log source data preempts the resources of the thread of the second target inspection log source data;

[0070] The batch-stream processing engine determines which data, higher in priority (greater than the default value), enters the batch-stream collection operation based on the priority of the inspection log source data. Low-priority data is forwarded to the data cache blocking queue for a delayed wait before the batch-stream collection operation proceeds. Dynamic priority interpolation can be introduced when synchronizing mixed batch-stream data, allowing high-priority data to be interpolated and queued while low-priority data is being processed.

[0071] The interpolation and queue grabbing process includes the following steps:

[0072] First, we design two queues, high-priority and low-priority. High-priority inspection log source data goes into the high-priority queue, while low-priority inspection log source data goes into the low-priority queue. The blocking queue scheduler is responsible for blocking queue data consumption and resource grabbing.

[0073] It should be noted that in principle, blocking and queuing can be performed when extracting, converting, and loading inspection log source data. In practice, blocking and queuing during data extraction and conversion have higher performance and cost-effectiveness because when the data has reached the loading step, the batch stream synchronization processing is nearly complete.

[0074] Secondly, if Figure 3 As shown, a spin mutex is defined to spin the high priority queue 31, triggering a blocking queue scheduler 32 (also known as a blocking queue grabber) to block the low priority queue 33, and at the same time start the high priority queue grabbing process 34, and obtain the currently running thread of the synchronous low priority data to seize resources 35;

[0075] Furthermore, if Figure 4 As shown in the figure, the resource seizure of high-priority inspection log source data may not succeed in one go, so it is necessary to preempt the resource multiple times between the high-priority thread and the low-priority thread.

[0076] Step S186: calling the batch-stream integrated processing engine to collect the first target inspection log source data.

[0077] In the embodiment of the present application, high-priority queue processing is used to enable high-priority data to be collected quickly without affecting the collection of low-priority data, thereby indirectly improving the efficiency of data collection.

[0078] In some embodiments, before step S184, the method further includes the following steps:

[0079] Step S1831: Aggregate and classify at least one first inspection log source data in the first queue to obtain a level category of each first inspection log source data in the first queue;

[0080] Step S1832: determining the number of queue grabbing times and queue grabbing duration of each first inspection log source data based on the level category of each first inspection log source data in the first queue;

[0081] Correspondingly, step S184 may include: triggering resource preemption processing of at least one first inspection log source data in the first queue based on the number of preemption times and preemption duration of each first inspection log source data in the first queue.

[0082] In order to improve the efficiency of the team grabbing process and avoid the occurrence of mistaken team grabbing as much as possible, it is necessary to first aggregate and classify the high-priority inspection log source data currently participating in the team grabbing operation, and perform batch flow team grabbing operations according to the different level categories of the inspection log source data. Assuming that there are n level categories in this team grabbing operation, the number of levels is k. This C(n,k) is used as the maximum number of parallel batch queue cycles. The data feature distances of different inspection log sources are calculated as classification indicators. The data feature distance d(x,y) can be expressed as: where x i ,y i It can represent the characteristic attributes of data. After calculating the characteristic distance, sort them according to the increasing and decreasing relationship, and then select the first j points with the smallest characteristic distance and count the frequency of occurrence of the category they belong to. Finally, the category with the highest frequency is used as the basis for predicting the classification. At the same time, in order to minimize the occupation of system resources by the queue-grabbing operation, the lower the high priority level x, the fewer times n it is allowed to grab the queue, and the shorter the allocated queue-grabbing duration t. The number of times decreases linearly with the decrease of priority level, which is expressed as n=z+x (where z is a positive integer greater than the minimum level), and the time t decreases logarithmically with the decrease of priority level, which is expressed as t=log z x, when the number of team grabbing times is completed or the allocated rotation team grabbing time expires and the team grabbing is not successful, the low-priority resources are released to allow it to continue running.

[0083] In the embodiment of the present application, compared with directly pulling or receiving data from the source data, when resources are limited and a large amount of data is waiting to be collected, this dynamic priority batch and stream integrated method can enable high-priority data to be collected quickly without affecting the collection of low-priority data, thereby indirectly improving the efficiency of data collection; by limiting the number of queue grabbing times and the duration of queue grabbing according to the level of the data, the efficiency of queue grabbing can be further improved and the occurrence of erroneous queue grabbing can be avoided as much as possible during queue grabbing.

[0084] Figure 5 This is a flow chart of a log collection method according to an embodiment of the present application. Figure 5 As shown, the method includes:

[0085] Step S501: Pull or receive inspection log source data according to basic configuration information of the device;

[0086] Among them, the basic configuration information can be the device name, IP (Internet Protocol) address, device model, software version, port number and port type, etc., and the inspection log source data of the corresponding device can be obtained from the database 51, file server 52, and message queue 53 of the network cloud inspection platform based on the basic configuration information of the device.

[0087] Step S502: Determine the priority of the inspection log source data; if the priority is low, execute step S503; if the priority is high, execute step S504;

[0088] Step S503: forwarding the inspection log source data to the cache blocking queue;

[0089] Step S504: calling the batch-stream integrated processing engine to process the inspection log source data;

[0090] Step S505: Determine whether it is a batch processing task or a stream processing task; if it is a stream processing task, execute step S506; if it is a batch processing task, execute step S507;

[0091] Step S506: call flow processing;

[0092] Step S507: calling batch processing;

[0093] Step S508: dynamic priority queue processing;

[0094] Step S509: Record the processed inspection log data.

[0095] In some embodiments, the method further includes the following steps S110 to S116:

[0096] Step S110: pre-processing the inspection log data of at least one device to obtain classified inspection log data corresponding to each type of device;

[0097] like Figure 6 As shown, after the log collection module 611 of the log correlation analysis engine 61 collects inspection log data from the non-relational database, file server, or messaging platform of the automatic inspection platform (also known as the cloud inspection platform) 62, the inspection log data is transferred to the log preprocessing module 612. The log preprocessing module 612 can perform operations such as cleaning, integration, transformation, and specification on the inspection log data. Inspection logs can also be classified and merged by device type to obtain classified inspection log data, which can reduce data storage costs to a certain extent.

[0098] Step S112: analyzing the classified inspection log data of at least one type of equipment to obtain at least one set of association rules between the classified inspection log data of at least one type of equipment;

[0099] like Figure 6 As shown, the log analysis module 613 can read the classified inspection log data from the database after log preprocessing, adopt the H-mine frequent pattern mining algorithm idea, transplant the traditional single-machine algorithm to the distributed computing environment, and perform log association rule mining based on distributed parallel computing.

[0100] Step S114: determining the association relationship between the classified inspection log data of at least one type of equipment based on the at least one set of association rules;

[0101] Step S116: Send the association relationship to the cloud inspection platform, and the cloud inspection platform associates and presents the classified inspection log data based on the association relationship.

[0102] like Figure 6 As shown, the result aggregation module 614 can analyze the correlation between the classified inspection log data of different types of equipment based on the association rules obtained through the distributed parallel algorithm, find out the rules and pattern characteristics of the simultaneous occurrence of abnormalities in various inspection items between different equipment types, and send the correlation between the inspection log data to the database of the automatic inspection platform 62 to provide strong data support for multi-dimensional correlation visualization, equipment disaster recovery, and intelligent operation and maintenance of the inspection log data.

[0103] In the embodiments of this application, a mechanism combining batch and stream integration with dynamic priority adjustment is proposed to optimize log data collection efficiency, reduce data latency, and enable high-priority data to be quickly transferred to the log preprocessing and log analysis processes. A personalized log data preprocessing mechanism is introduced to reduce dirty data and improve the accuracy of analysis results. An association mining algorithm based on a distributed computing environment is proposed, and the optimization of batch and stream parallel processing improves rule mining efficiency. This algorithm fully utilizes the advantages of a distributed environment, has better performance and scalability, and can more efficiently analyze and mine large-scale data.

[0104] In some embodiments, step S110 of "pre-processing the inspection log data of at least one device to obtain classified inspection log data corresponding to each type of device" can be implemented by the following steps S1101 to S1103:

[0105] Step S1101: determining whether each inspection log data of each device complies with the corresponding inspection item rules of the corresponding type of device;

[0106] Among them, such as Figure 7 As shown, custom rule filter 70 (inspection log filter policy master adapter) is serially connected to adapt inspection log filtering rules for each type of device. Inspection log filter policy master adapter 70 adapts to server log filter adapter 71, router log filter adapter 72, switch log filter adapter 73, firewall log filter adapter 74, and WAF (Web Application Firewall) log filter adapter 75. Server log filter adapter 71 can also adapt to host log filter 711, physical server log filter 712, and storage node server log filter 713.

[0107] Each type of device adapter is adapted to the inspection item rules under the corresponding type of device. For example, the host log filter 711 adapts to inspection item rules such as memory size inspection rules and disk size inspection rules; the physical server log filter 712 adapts to power status inspection rules and system time inspection rules; the storage node server log filter 713 adapts to system version inspection rules and SELinux status inspection rules; the router log filter adapter 72 adapts to IP status inspection rules and fan movement status inspection rules; the switch log filter adapter 73 adapts to temperature inspection rules and CPU usage inspection rules; the firewall log filter adapter 74 adapts to power status inspection rules and alarm information inspection rules; the WAF log filter adapter 75 adapts to memory usage inspection rules and device temperature inspection rules.

[0108] This chain adaptation filtering method can match the pattern of each inspection item log data to the greatest extent, has a good effect on cleaning the inspection data, and effectively reduces the existence of dirty data.

[0109] Step S1102: When each inspection log data complies with the corresponding inspection item rule of the corresponding type of equipment, the corresponding inspection log data is subjected to standard normalization processing to obtain corresponding standard inspection log data;

[0110] Step S1103: Based on the device type of the at least one device, the standard inspection log data of the at least one device is classified and merged to obtain classified inspection log data corresponding to each type of device.

[0111] In an embodiment of the present application, by supporting personalized custom filtering rule configuration for the inspection log of each type of equipment inspection item, the dirty data in each inspection item log can be reduced, and the data that meets the rules is then normalized. At the same time, classifying and merging the inspection logs by equipment type can reduce the cost of storing data to a certain extent.

[0112] Figure 8 This is a flow chart of a log preprocessing method according to an embodiment of the present application. Figure 8 As shown, the method includes the following steps:

[0113] Step S801: Start;

[0114] Step S802: Select equipment inspection items;

[0115] Step S803: configuring personalized filtering rules for device inspection items;

[0116] Among them, Figure 7 As shown, you can configure personalized filtering rules for the inspection items of each type of device.

[0117] Step S804: Determine whether the inspection log data meets the personalized filtering rules. If so, execute step S805; if not, execute step S807;

[0118] Among them, it can be judged whether each inspection log data of each device complies with the corresponding inspection item rules of the corresponding type of device. For example, if the inspection log data of the device is the IP status of the router, it can be judged whether the IP status of the router complies with the IP status inspection rules of the router log filter adapter 72.

[0119] Step S805: performing standard normalization processing on the inspection log data that meets the filtering rules;

[0120] Step S806: Classify and merge the inspection log data after standard normalization processing;

[0121] Among them, classifying and merging inspection logs by device type can reduce the cost of storing data to a certain extent.

[0122] Step S807: End.

[0123] In some embodiments, step S112 of "analyzing the classified inspection log data of at least one type of device to obtain at least one set of association rules between the classified inspection log data of at least one type of device" may be implemented by the following steps S1121 and S1122:

[0124] Step S1121: Based on a parallel association analysis algorithm, mining and analyzing the classified inspection log data of at least one type of equipment is performed to determine frequent item sets of the classified inspection log data of the at least one type of equipment;

[0125] Step S1122: determining at least one set of association rules between classified inspection log data of at least one type of equipment based on the frequent itemsets;

[0126] The parallel association analysis algorithm is used to convert data into data structure storage and mine frequent item sets.

[0127] Among them, the parallel association analysis algorithm is also called a distributed parallel algorithm. The distributed parallel algorithm is a memory-based, efficient frequent item set mining algorithm. At the same time, based on the H-struct, its node storage sub-unit uses an array and a linked list plus a weighted directed graph. It can traverse data faster through weight values ​​and save more space than traditional storage methods. Figure 9 This is a diagram combining arrays and linked lists in H-struct, such as Figure 9 As shown, h i Indicates the size of the weight. Figure 10This is the structure diagram of the array and linked list in H-struct; Figure 10 As shown in Figure 2, by partitioning the data, only one partition is mined at a time and the results are finally integrated.

[0128] Distributed parallel processing can include parallelization of data conversion into data structure storage and parallelization of mining frequent itemsets.

[0129] Parallel data conversion into data structure storage: Before frequent itemset mining, the transaction set must be traversed to convert the data into a data structure for storage. Frequent item projections are then obtained in parallel. The data is then transformed using the frequent item projections and the frequent item permutations. The item ID (identity document) and connection pointer for each frequent item are stored in a new data structure. To obtain frequent item projections in parallel, the slicing function is first called to slice each row. Then, a filtering function is called to select items with a support count not less than the minimum support count. Finally, the aggregate function is used to return the result as the frequent item projection. The frequent item set results and the frequent item projections are encapsulated in a new data structure.

[0130] Figure 11 This is a flowchart of a method for converting data into a data structure for storage and parallel processing according to an embodiment of the present application. Figure 11 As shown, the method includes the following steps:

[0131] Step S1101: Start;

[0132] Step S1102: input data;

[0133] Step S1103: flatten the data using a slicing function;

[0134] Step S1104: using a filtering function to filter and obtain frequent item projections;

[0135] Step S1105: Connect the same frequent items, and use a mapping function to connect the frequent set results and the frequent item projections to a new data structure;

[0136] Step S1106: the aggregation function returns the result;

[0137] Step S1107: End.

[0138] Parallelization of frequent item set mining: In the distributed parallel mining algorithm, transactions are partitioned according to the first item data. After processing the transaction set of a partition, the transaction set of the partition needs to be repartitioned until all partitions are traversed. Figure 12a As shown in the figure, the stand-alone algorithm in the related art is limited by hardware resources. Processing transactions in one queue at a time can reduce the memory requirements, but it also sacrifices time while saving memory space. Figure 12b As shown, distributed computing environments eliminate the hardware resource limitations of a single machine, enabling transactions from multiple queues to be processed simultaneously through a map / reduce process. Leveraging the advantages of clusters, trading space for time, each queue receives the required set of transactions at once.

[0139] Figure 13 This is a flow chart of a log analysis method according to an embodiment of the present application. Figure 13 As shown, the method includes the following steps:

[0140] Step S1301: Start;

[0141] Step S1302: importing the inspection log data set from the pre-processed database;

[0142] The inspection log data set includes classified inspection log data after classification and merging.

[0143] Step S1303: using a parallel association analysis algorithm to mine and analyze the inspection log data to obtain frequent item sets;

[0144] The parallel association analysis algorithm is also called a frequent itemset mining algorithm.

[0145] Step S1304: Determine whether the frequent itemsets meet the index requirements. If so, execute step S1305; if not, execute step S1302;

[0146] Step S1305: Analyze association rules between inspection log data based on the frequent item sets;

[0147] Step S1306: Determine whether the association rule meets the index requirements. If yes, execute step S1307; if not, execute step S1305.

[0148] Step S1307: Output the result;

[0149] Among them, association rules that meet the index requirements can be output.

[0150] Step S1308: End.

[0151] The embodiment of the present application proposes to introduce a weighted directed graph based on the existing data structure to optimize query efficiency and reduce storage space. At the same time, based on the distributed computing environment, by converting data into data structure storage parallelism and mining frequent item sets parallelism, it solves the problem of low efficiency of traditional single-machine association rule mining algorithms, helps to improve the speed of association rule mining, and fully utilizes the high performance characteristics of distributed computing multiple nodes. It also has good performance and scalability in the face of large-scale data growth.

[0152] In some embodiments, step S114 “analyzing the association relationship between the classified inspection log data of at least one type of device based on the at least one set of association rules” may be implemented by the following steps S1141 and S1142:

[0153] Step S1141: Filtering a first association rule that satisfies the minimum support and the minimum confidence from the at least one group of association rules;

[0154] The first association rule is also called a strong association rule.

[0155] Step S1142: Based on the first association rule, determine the association relationship between the classified inspection log data of at least one type of equipment.

[0156] Through distributed parallel mining, strong association rules that meet minimum support and confidence levels can be identified. Based on these strong association rules, the correlations between inspection logs of different types of equipment can be analyzed, identifying patterns and characteristics of the simultaneous occurrence of abnormalities in various inspection items across different devices. The rapid mining capabilities of the distributed parallel association algorithm allow for the immediate placement of the rule results in a buffer queue after a strong association rule is discovered. This allows for rapid log data association analysis and aggregation without having to wait for all rules to be mined before analysis can begin, speeding up log data analysis and presentation. This provides strong data support for multi-dimensional log data association visualization, equipment disaster recovery, and intelligent operations and maintenance.

[0157] In some embodiments, the method further includes the following steps S116 to S122:

[0158] Step S116: Binding each set of the association rules with the corresponding classified inspection log data;

[0159] Step S118: determining a second association rule other than the first association rule in at least one group of the association rules; the association degree between different types of inspection log data corresponding to the first association rule is higher than the association degree between different types of inspection log data corresponding to the second association rule;

[0160] The first association rule may be a strong association rule, and the second association rule may be a weak association rule.

[0161] Step S120: updating the dynamic adjustment factor of the corresponding classified inspection log data based on the first association rule and the second association rule respectively;

[0162] Step S122: Based on the similarity between the inspection log source data of at least one device, the device link urgency, and the updated dynamic adjustment factor, the priority of the inspection log source data of the at least one device is updated.

[0163] It should be noted that association rules and historical data can be recorded in a distributed search engine at the same time, the relationship between association rules and historical data can be regressed, and a dynamic factor model can be recorded. The process is as follows:

[0164] Association rules and data binding relationship: recursively poll the inspection log analysis results, bind each set of association rules with the inspection log data, and record the relationship in the distributed engine. The rule p mapping relationship can be expressed as P(a1, a2, ... a n ), in fact, a1, a2, ... a n Corresponding inspection log data.

[0165] Filter and group strong rule relationships and weak rule relationships and perform statistical tracing: Group and count the corresponding relationships between strong rules and weak rules and enter them into different indexes.

[0166] Adjust the dynamic factors of the inspection items mapped by strong rule relationships and weak rule relationships respectively: In addition to the strong and weak rule m, the inspection item log data delay n is also one of the important factors. It is increased by one for each waiting time and reduced by one for each execution time. The strong and weak association rules are used as the main factors, and the delay is used as the secondary factor. The dynamic adjustment factor D = am + n, such as Figure 6 As shown, after the result aggregation module 614 calculates the dynamic adjustment factor, it forwards the dynamic adjustment factor back to the log collection module 611 through the message queue as one of the factors affecting the collection priority.

[0167] In an embodiment of the present application, the priority of the inspection log data is updated based on association rules, so that the priority of the inspection log data can be dynamically updated, the collection efficiency of the inspection log data can be improved, the value of the inspection log data can be mined, and abnormal inspection log data alarms can be issued in a timely manner.

[0168] The association analysis method for network cloud inspection logs proposed in the embodiment of the present application adopts batch-stream integration and a dynamic perception mechanism of log data priority to optimize log collection efficiency. Log preprocessing supports personalized configuration rules to clean and standardize log data to improve the accuracy of analysis results. At the same time, it is based on a distributed parallel association analysis algorithm to more efficiently analyze and mine large-scale data.

[0169] The embodiment of the present application provides a network cloud inspection log correlation analysis device 61, such as Figure 6As shown, the device 61 can be divided into four core modules: a log collection module 611, a log preprocessing module 612, a log analysis module 613, and a result aggregation module 614. The inspection log data source is the network cloud automated inspection platform 62. The log collection module 611 synchronizes the inspection log data from the inspection platform 62. The log preprocessing module 612 performs data standard normalization processing to obtain rule-structured data that meets the requirements of the log analysis module 613. The log association analysis algorithm is trained to calculate the rule-structured data and analyze the result to obtain an association model. The result aggregation module 614 aggregates the inspection results and presents them visually.

[0170] The embodiments of this application provide a complete set of detailed processes and solutions for network cloud inspection log correlation analysis; propose a mechanism that combines batch and stream integration with dynamic priority adjustment to optimize log data collection efficiency, reduce data latency, and quickly transfer high-priority data to log preprocessing and log analysis processes; propose the introduction of a personalized log data preprocessing mechanism to reduce dirty data and improve the accuracy of analysis results. A correlation mining algorithm based on a distributed computing environment is proposed, and the optimization idea of ​​batch and stream parallel processing improves the efficiency of rule mining; fully utilizing the advantages of a distributed environment, it has better performance and scalability, thereby more efficiently analyzing and mining large-scale data.

[0171] It should be noted that, in the embodiment of the present application, if the above-mentioned inspection log collection method is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the relevant technology can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable an electronic device (which can be a mobile phone, tablet computer, desktop computer, personal digital assistant, navigator, digital phone, video phone, television, sensor device, etc.) to execute all or part of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a U disk, a mobile hard disk, a read-only memory (ROM), a magnetic disk or an optical disk. In this way, the embodiment of the present application is not limited to any specific combination of hardware and software.

[0172] Based on the foregoing embodiments, an embodiment of the present application provides an inspection log collection device, which includes the various modules included and can be implemented by a processor in the device; of course, it can also be implemented by a specific logic circuit; in the implementation process, the processor can be a central processing unit (CPU), a microprocessor (MPU), a digital signal processor (DSP), or a field programmable gate array (FPGA), etc.

[0173] Figure 14 This is a schematic diagram of the composition structure of an inspection log collection device according to an embodiment of the present application. Figure 14 As shown, the apparatus 1400 includes an acquisition module 1401, a judgment module 1402, a first determination module 1403, and a collection module 1404, wherein:

[0174] The acquisition module 1401 is used to obtain the inspection log source data of at least one device from the cloud inspection platform based on the batch and stream integration mechanism;

[0175] A determination module 1402 is configured to determine the priority of the inspection log source data of the at least one device;

[0176] A first determining module 1403 is configured to determine a collection order of at least one of the inspection log source data based on a priority of at least one of the inspection log source data;

[0177] The collection module 1404 is configured to call the batch-stream integrated processing engine to collect at least one inspection log source data based on the collection order to obtain at least one inspection log data.

[0178] In some embodiments, the judgment module 1402 includes: a first determination submodule, used to determine the similarity between the inspection log source data of at least one device, the device link urgency and the dynamic adjustment factor; the device link urgency is used to characterize the urgency of collecting the inspection log source data of the link where the device is located; the dynamic adjustment factor is used to adjust the priority of the inspection log source data of the device; a second determination submodule, used to determine the priority of the inspection log source data of at least one device based on the similarity between the inspection log source data of at least one device, the device link urgency and the dynamic adjustment factor.

[0179] In some embodiments, the first determination submodule includes: a first determination unit, used to determine the data attribute integrity, data volume integrity and upper and lower level chain integrity of the inspection log source data of at least one device; a second determination unit, used to determine the link integrity of each device based on the data attribute integrity, data volume integrity and upper and lower level chain integrity of each device; a third determination unit, used to determine the device link urgency of each device based on the link integrity of each device and a preset mapping relationship.

[0180] In some embodiments, the priority includes a first priority and a second priority, the first priority is higher than the second priority, and the acquisition module 1404 includes: a placement submodule, used to place at least one first inspection log source data of the first priority into the first queue, and place at least one second inspection log source data of the second priority into the second queue; the first queue is a mutex lock or spin lock queue; a queue grabbing submodule, used to trigger at least one first inspection log source data in the first queue to perform resource grabbing processing when calling the batch-stream integrated processing engine to collect the second target inspection log source data in the second queue, so that the first target inspection log source data grabs the resources of the thread of the second target inspection log source data; the acquisition submodule, used to call the batch-stream integrated processing engine to collect the first target inspection log source data.

[0181] In some embodiments, the acquisition module 1404 further includes:

[0182] A classification submodule is used to aggregate and classify at least one first inspection log source data in the first queue to obtain a level category of each first inspection log source data in the first queue; a third determination submodule is used to determine the number of times each first inspection log source data is preempted and the length of time it is preempted based on the level category of each first inspection log source data in the first queue; correspondingly, the preemption submodule includes: a preemption unit, which is used to trigger resource preemption processing of at least one first inspection log source data in the first queue based on the number of times each first inspection log source data is preempted and the length of time it is preempted.

[0183] In some embodiments, the device also includes: a preprocessing module, which is used to preprocess the inspection log data of at least one device to obtain classified inspection log data corresponding to each type of device; a first analysis module, which is used to analyze the classified inspection log data of at least one type of device to obtain at least one set of association rules between the classified inspection log data of at least one type of device; a second analysis module, which is used to determine the association relationship between the classified inspection log data of at least one type of device based on the at least one set of association rules; and a sending module, which is used to send the association relationship to the cloud inspection platform, and the cloud inspection platform associates and presents the classified inspection log data based on the association relationship.

[0184] In some embodiments, the preprocessing module includes: a judgment submodule, used to judge whether each inspection log data of each of the devices complies with the corresponding inspection item rules of the corresponding type of device; a processing submodule, used to perform standard normalization processing on the corresponding inspection log data when each of the inspection log data complies with the corresponding inspection item rules of the corresponding type of device, so as to obtain corresponding standard inspection log data; a merging submodule, used to classify and merge the standard inspection log data of at least one device based on the device type of at least one device, so as to obtain classified inspection log data corresponding to each type of device.

[0185] In some embodiments, the first analysis module includes: a first analysis submodule, which is used to mine and analyze the classified inspection log data of at least one type of equipment based on a parallel association analysis algorithm, and determine the frequent item sets of the classified inspection log data of at least one type of equipment; a fourth determination submodule, which is used to determine at least one set of association rules between the classified inspection log data of at least one type of equipment based on the frequent item sets; wherein, the parallel association analysis algorithm is used to realize the conversion of data into data structure storage and mining of frequent item sets.

[0186] In some embodiments, the second analysis module includes: a screening submodule for screening a first association rule that satisfies the minimum support and minimum confidence from the at least one group of association rules; and a second analysis submodule for determining the association relationship between the classified inspection log data of at least one type of equipment based on the first association rule.

[0187] In some embodiments, the device also includes: a binding module for binding each group of the association rules and the corresponding classified inspection log data; a second determination module for determining a second association rule other than the first association rule in at least one group of the association rules; the degree of association between different types of inspection log data corresponding to the first association rule is higher than the degree of association between different types of inspection log data corresponding to the second association rule; a first update module for updating the dynamic adjustment factor of the corresponding classified inspection log data based on the first association rule and the second association rule respectively; a second update module for updating the priority of the inspection log source data of at least one device based on the similarity between the inspection log source data of at least one device, the device link urgency and the updated dynamic adjustment factor.

[0188] The description of the above device embodiment is similar to the description of the above method embodiment and has similar beneficial effects as the method embodiment. For technical details not disclosed in the device embodiment of this application, please refer to the description of the method embodiment of this application for understanding.

[0189] Correspondingly, an embodiment of the present application provides a device, Figure 15 This is a hardware entity diagram of an electronic device according to an embodiment of the present application, such as Figure 15 As shown, the hardware entity of the electronic device 1500 includes: a memory 1501 and a processor 1502, the memory 1501 stores a computer program that can be run on the processor 1502, and the processor 1502 implements the steps in the inspection log collection method of the above embodiment when executing the program.

[0190] The memory 1501 is configured to store instructions and applications executable by the processor 1502, and can also cache data to be processed or processed by the processor 1502 and the various modules in the device 1500 (for example, image data, audio data, voice communication data and video communication data), which can be implemented through flash memory (FLASH) or random access memory (RAM).

[0191] Correspondingly, an embodiment of the present application provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps in the inspection log collection method provided in the above embodiment are implemented.

[0192] It should be noted that the description of the above storage medium and device embodiments is similar to the description of the above method embodiments, and has similar beneficial effects as the device embodiments. For technical details not disclosed in the storage medium and method embodiments of this application, please refer to the description of the device embodiments of this application for understanding.

[0193] It should be understood that "one embodiment" or "an embodiment" mentioned throughout the specification means that the specific features, structures or characteristics related to the embodiment are included in at least one embodiment of the present application. Therefore, "in one embodiment" or "in an embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. In addition, these specific features, structures or characteristics can be combined in one or more embodiments in any suitable manner. It should be understood that in the various embodiments of the present application, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application. The above-mentioned serial numbers of the embodiments of the present application are for description only and do not represent the advantages and disadvantages of the embodiments.

[0194] It should be noted that, in this document, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or apparatus comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or apparatus comprising the element.

[0195] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0196] The units described above as separate components may or may not be physically separate, and the components displayed as units may or may not be physical units; they may be located in one place or distributed across multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. In addition, the functional units in the various embodiments of the present application may all be integrated into one processing unit, or each unit may be separately used as a unit, or two or more units may be integrated into one unit; the above-mentioned integrated units may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0197] Those skilled in the art will appreciate that all or part of the steps in implementing the above-mentioned method embodiments can be accomplished by hardware associated with program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, the steps of the above-mentioned method embodiments are executed; and the aforementioned storage medium includes various media that can store program codes, such as a mobile storage device, a read-only memory (ROM), a magnetic disk, or an optical disk. Alternatively, if the above-mentioned integrated unit of the present application is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application, or the part that contributes to the relevant technology, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for causing a computer device (which can be a mobile phone, tablet computer, desktop computer, personal digital assistant, navigator, digital phone, video phone, television, sensor device, etc.) to execute all or part of the methods described in each embodiment of the present application. And the aforementioned storage medium includes various media that can store program codes, such as a mobile storage device, a ROM, a magnetic disk, or an optical disk.

[0198] The methods disclosed in the several method embodiments provided in this application can be arbitrarily combined, if they do not conflict, to obtain new method embodiments. The features disclosed in the several product embodiments provided in this application can be arbitrarily combined, if they do not conflict, to obtain new product embodiments. The features disclosed in the several method or device embodiments provided in this application can be arbitrarily combined, if they do not conflict, to obtain new method embodiments or device embodiments.

[0199] The above is merely an embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

Claims

1. A method for collecting inspection logs, characterized in that: The method comprises: Based on the batch-stream integration mechanism, obtain the inspection log source data of at least one device from the cloud inspection platform; Determining the priority of the inspection log source data of the at least one device; determining a collection order of at least one of the inspection log source data based on a priority of at least one of the inspection log source data; Based on the collection order, calling the batch-stream integrated processing engine to collect at least one of the inspection log source data to obtain at least one inspection log data; Preprocessing the inspection log data of at least one device to obtain classified inspection log data corresponding to each type of device; Analyzing classified inspection log data of at least one type of equipment to obtain at least one set of association rules between the classified inspection log data of at least one type of equipment; Determining, based on the at least one set of association rules, an association relationship between classified inspection log data of at least one type of equipment; Sending the association relationship to the cloud inspection platform, and the cloud inspection platform presents the classified inspection log data in an associated manner based on the association relationship; Among them, the priority includes a first priority and a second priority, the first priority is higher than the second priority, and the calling of the batch-stream integrated processing engine to collect at least one of the inspection log source data includes: putting at least one first inspection log source data of the first priority into the first queue, and putting at least one second inspection log source data of the second priority into the second queue; the first queue is a mutex or spin lock queue; when calling the batch-stream integrated processing engine to collect the second target inspection log source data, triggering at least one first inspection log source data in the first queue to perform resource preemption processing, so that the first target inspection log source data preempts the resources of the thread of the second target inspection log source data; calling the batch-stream integrated processing engine to collect the first target inspection log source data.

2. The method according to claim 1, characterized in that The determining the priority of the inspection log source data of the at least one device includes: Determining the similarity between inspection log source data of at least one device, the device link urgency, and a dynamic adjustment factor; the device link urgency is used to indicate the urgency of collecting inspection log source data of the link where the device is located; the dynamic adjustment factor is used to adjust the priority of the inspection log source data of the device; The priority of the inspection log source data of the at least one device is determined based on the similarity between the inspection log source data of the at least one device, the device link urgency, and the dynamic adjustment factor.

3. The method according to claim 1, characterized in that The preprocessing of the inspection log data of at least one device to obtain classified inspection log data corresponding to each type of device includes: Determine whether each inspection log data of each of the devices complies with the corresponding inspection item rules of the corresponding type of device; When each inspection log data complies with the corresponding inspection item rule of the corresponding type of equipment, performing standard normalization processing on the corresponding inspection log data to obtain corresponding standard inspection log data; Based on the device type of at least one device, the standard inspection log data of at least one device is classified and merged to obtain classified inspection log data corresponding to each type of device.

4. The method according to claim 1, wherein The analyzing of the classified inspection log data of at least one type of equipment to obtain at least one set of association rules between the classified inspection log data of at least one type of equipment includes: Based on a parallel association analysis algorithm, mining and analyzing the classified inspection log data of at least one type of equipment are performed to determine frequent item sets of the classified inspection log data of the at least one type of equipment; Determining at least one set of association rules between classified inspection log data of at least one type of equipment based on the frequent item sets; The parallel association analysis algorithm is used to convert data into data structure storage and mine frequent item sets.

5. The method according to claim 4, characterized in that After determining the association relationship between the classified inspection log data of at least one type of equipment, the method further includes: Binding each set of association rules to corresponding classified inspection log data; Determine a second association rule other than the first association rule in at least one group of the association rules; the association degree between different types of inspection log data corresponding to the first association rule is higher than the association degree between different types of inspection log data corresponding to the second association rule; updating the dynamic adjustment factor of the corresponding classified inspection log data based on the first association rule and the second association rule respectively; Based on the similarity between the inspection log source data of at least one device, the device link urgency, and the updated dynamic adjustment factor, the priority of the inspection log source data of the at least one device is updated.

6. A patrol log collection device, characterized in that: The device comprises: The acquisition module is used to obtain the inspection log source data of at least one device from the cloud inspection platform based on the batch and stream integration mechanism; a judgment module, configured to judge the priority of the inspection log source data of the at least one device; A first determining module is configured to determine a collection order of at least one of the inspection log source data based on a priority of at least one of the inspection log source data; the priority includes a first priority and a second priority, the first priority being higher than the second priority; An acquisition module is used to call the batch-stream integrated processing engine to collect at least one of the inspection log source data based on the acquisition order to obtain at least one inspection log data; the acquisition module includes: a placement submodule, used to place at least one first inspection log source data of the first priority into a first queue, and place at least one second inspection log source data of the second priority into a second queue; the first queue is a mutex lock or spin lock queue; a queue grabbing submodule is used to trigger at least one first inspection log source data in the first queue to perform resource grabbing processing when calling the batch-stream integrated processing engine to collect the second target inspection log source data, so that the first target inspection log source data grabs the resources of the thread of the second target inspection log source data; the acquisition submodule is used to call the batch-stream integrated processing engine to collect the first target inspection log source data; A preprocessing module, configured to preprocess the inspection log data of at least one device to obtain classified inspection log data corresponding to each type of device; A first analysis module is configured to analyze classified inspection log data of at least one type of equipment to obtain at least one set of association rules between the classified inspection log data of at least one type of equipment; A second analysis module is configured to determine, based on the at least one set of association rules, an association relationship between classified inspection log data of at least one type of equipment; The sending module is used to send the association relationship to the cloud inspection platform, and the cloud inspection platform associates and presents the classified inspection log data based on the association relationship.

7. An electronic device comprising a memory and a processor, wherein the memory stores a computer program that can be run on the processor, characterized in that: When the processor executes the program, the steps of the inspection log collection method according to any one of claims 1 to 5 are implemented.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the inspection log collection method according to any one of claims 1 to 5 are implemented.

Images