Log data fusion processing method and device

Abstract

The embodiment of the specification discloses a log data fusion processing method, which is used for efficiently processing log data. The method comprises the following steps: determining a log processing interval according to the log processing requirement of a power transmission equipment system; dividing the log processing interval to obtain a plurality of preset time periods; acquiring real-time logs generated by the power transmission equipment system in the preset time periods, and determining batch logs corresponding to the real-time logs in the power transmission equipment system; filtering repeated data in the real-time logs and the batch logs, and determining missing data of each preset time period of the power transmission equipment system; determining abnormal conditions in the power transmission equipment system based on the missing data of each preset time period, so as to complete the fusion processing of the log data.

Description

TECHNICAL FIELD

[0001] The present specification relates to the technical field of data processing, and in particular, to a log data fusion processing method and device. BACKGROUND

[0002] Logs, as important records of device running states, are an important source for analyzing the running states of field devices. In order to balance real-time performance and data comprehensiveness, current industrial devices use real-time logs plus full logs. With the rapid development of the Internet, the Internet of Things, cloud computing, and the like, data scales become larger and larger, and contents become more and more complex. Massive log processing is an important part of data processing.

[0003] In the prior art, massive log data is processed based on a message queue. After stream data and batch data are all imported into the message queue, the stream batch data is processed separately or in an integrated manner. The above-mentioned methods all need to process all batch data and stream data, which causes excessive computing pressure in log processing and results in waste of a large amount of computing resources.

[0004] Therefore, there is a need for an efficient log data fusion processing method. SUMMARY

[0005] One or more embodiments of the present specification provide a log fusion processing method and device, which are used to solve the technical problem of how to provide an efficient log fusion processing method.

[0006] One or more embodiments of the present specification adopt the following technical solutions:

[0007] One or more embodiments of the present specification provide a log data fusion processing method, which comprises:

[0008] According to log processing requirements of a power transmission device system, a log processing interval is determined, and the log processing interval is divided to obtain a plurality of preset time periods;

[0009] Real-time logs generated by the power transmission device system in a preset time period are obtained, and batch logs corresponding to the real-time logs are determined in the power transmission device system. The real-time logs are used to record real-time running data of each device component in the power transmission device system, and the batch logs are used to record historical running data to be processed in each device component in the power transmission device.

[0010] Repeated data in the real-time logs and the batch logs is filtered to determine missing data of each preset time period of the power transmission device system.

[0011] Determine the abnormal situation in the power transmission equipment system based on the missing data of each preset time period, so as to complete the fusion processing of the log data.

[0012] Optionally, in one or more embodiments of the present specification, the real-time log generated by the power transmission equipment system in a preset time period is obtained, and the batch log corresponding to the real-time log in the power transmission equipment system is determined, specifically comprising:

[0013] Obtain the full amount of log of the power transmission equipment system, wherein the full amount of log includes the to-be-processed real-time log and the to-be-processed batch log;

[0014] Determine the data of the to-be-processed real-time log in the preset time period as the real-time log;

[0015] According to the time stamp corresponding to the preset time period, divide the data corresponding to the time period of the real-time log in the to-be-processed batch log as the batch log.

[0016] Optionally, in one or more embodiments of the present specification, the repeated data in the real-time log and the batch log is filtered, and the missing data of each preset time period of the power transmission equipment system is determined, specifically comprising:

[0017] Perform a preset first processing on the data in the real-time log and the batch log to obtain a first mapping data set;

[0018] Group processing is performed on the data in the first mapping data set to obtain a plurality of second mapping data sets, and a mapping relationship between the first mapping data set and the second mapping data set is determined; wherein the data in the second mapping data set includes one or more of second mapping batch data and second mapping stream data;

[0019] If it is determined that the second mapping batch data is not equal to the second mapping stream data, the first missing data of each preset time period of the power transmission equipment system is determined according to the second mapping batch data;

[0020] If it is determined that the second mapping batch data is equal to the second mapping stream data, the second missing data of each preset time period of the power transmission equipment system is obtained according to the first mapping data set and the second mapping data set;

[0021] Fuse the first missing data and the second missing data to obtain the missing data of each preset time period of the power transmission equipment system.

[0022] Optionally, in one or more embodiments of the present specification, the real-time log generated by the power transmission equipment system in a preset time period is obtained, and the batch log corresponding to the real-time log in the power transmission equipment system is determined, specifically comprising:

[0023] determine an index field of the data according to a feature of the data in the real-time log and the batch log, wherein the feature comprises any one or more of the following: device running time, log collection time, device ID, log type;

[0024] perform mapping processing on the index field to map the index field to a corresponding first mapping data set, wherein the data in the first mapping data set comprises first mapping batch data and first mapping stream data.

[0025] Optionally, in one or more embodiments of the present specification, the grouping processing on the data in the first mapping data set to obtain a plurality of second mapping data sets, and determining a mapping relationship between the first mapping data set and the second mapping data set, specifically comprises:

[0026] performing modulo operation on the data in the first mapping data set according to a preset array to obtain a plurality of second mapping data sets;

[0027] determining the mapping relationship between the first mapping data set and the second mapping data set according to the modulo operation process, and storing the mapping relationship in a preset mapping set in the form of a programming language.

[0028] Optionally, in one or more embodiments of the present specification, if it is determined that the second mapping batch data is not equal to the second mapping stream data, the first missing data of each preset time period of the power transmission equipment system is determined according to the second mapping batch data, specifically comprising:

[0029] if the second mapping batch data is not in the second mapping stream data, determining the corresponding first mapping batch data of the second mapping batch data in the first mapping data set based on the mapping relationship between the first mapping data set and the second mapping data set;

[0030] obtaining batch data corresponding to the first mapping batch data in the batch log, and taking the batch data as the first missing data.

[0031] Optionally, in one or more embodiments of the present specification, if it is determined that the second mapping batch data is equal to the second mapping stream data, the second missing data of each preset time period of the power transmission equipment system is obtained according to the first mapping data set and the second mapping data set, specifically comprising:

[0032] screening the plurality of second mapping data sets to determine a first subset of the second mapping data sets, and obtaining a second subset of the second mapping data sets; wherein the first subset only contains second mapping stream data;

[0033] According to the mapping relationship between the first mapping data set and the second mapping data set, a third subset corresponding to the second subset is obtained; wherein, the third subset is located in the first mapping data set;

[0034] According to the plurality of second mapping data sets and the second subset, second missing data of the power transmission equipment system in each preset time period is obtained.

[0035] Optionally, in one or more embodiments of the present specification, the second missing data of the power transmission equipment system in each preset time period is obtained according to the first mapping data set and the second mapping data set, specifically including:

[0036] According to the difference set of the second mapping data set and the second subset, the missing batch data in the second mapping set is determined;

[0037] According to the mapping relationship, the corresponding first mapping batch data and the corresponding first mapping flow data of the missing batch data in the first mapping data set are determined; and according to the corresponding first mapping batch data and the corresponding first mapping flow data, a fourth subset is obtained;

[0038] The data in the third subset and the fourth subset are merged to obtain the second missing data of the power transmission equipment system in each preset time period.

[0039] Optionally, in one or more embodiments of the present specification, the abnormal situation in the power transmission equipment system is determined based on the missing data in each preset time period, specifically including:

[0040] According to the pre-stored data exception list, the abnormal reason corresponding to each data in the missing data is obtained;

[0041] According to the characteristics of the data, the equipment ID corresponding to each data in the missing data is determined, and the equipment model is determined based on the equipment ID;

[0042] The abnormal reason, the preset time period and the equipment model are input into the pre-trained deep learning network to output the equipment parameter adjustment strategy;

[0043] The equipment parameter adjustment strategy is traced back to the equipment based on the equipment ID to adjust the running state of the equipment.

[0044] One or more embodiments of the present specification provide a log data fusion processing device, comprising:

[0045] at least one processor; and,

[0046] a memory in communication connection with the at least one processor; wherein,

[0047] The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to:

[0048] According to the log processing requirement of the power transmission equipment system, a log processing interval is determined, and the log processing interval is divided to obtain a plurality of preset time periods;

[0049] Real-time logs generated by the power transmission equipment system in a preset time period are obtained, and batch logs corresponding to the real-time logs are determined in the power transmission equipment system;

[0050] Repeated data in the real-time logs and the batch logs is filtered, and missing data of each preset time period of the power transmission equipment system is determined;

[0051] Based on the missing data of each preset time period, an abnormal situation in the power transmission equipment system is determined to complete the fusion processing of the log data.

[0052] The above at least one technical solution adopted by the embodiments of the present specification can achieve the following beneficial effects:

[0053] The embodiments of the present specification realize batch and real-time integrated processing mode by fusing batch logs and real-time logs, filter out repeated data in the front end, and reduce the pressure of subsequent process processing. According to the time sequence characteristics of the log data, the time is sliced according to the preset time period, so that the data in each preset time period can be processed respectively, and the size of the preset time period can be adjusted according to the demand, so that the data fusion processing process is more flexible. The abnormal equipment is obtained in real time through the positioning of the missing data, which is convenient for comprehensive diagnosis of the equipment operation and reduces the operation and maintenance cost of the equipment. BRIEF DESCRIPTION OF DRAWINGS

[0054] In order to more clearly illustrate the technical solutions in the embodiments of the present specification or the prior art, the drawings needed to be used in the embodiments or prior art description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments described in the present specification, and other drawings can be obtained by those skilled in the art without creative labor. In the drawings:

[0055] Figure 1 A flowchart of a log data fusion processing method provided by the embodiments of the present specification;

[0056] Figure 2 An internal structure diagram of a log data fusion processing device provided by the embodiments of the present specification. DETAILED DESCRIPTION

[0057] The embodiment of the present specification provides a log data fusion processing method and device.

[0058] The log is an important state time of system running, which is generated by each module and component during system running, and is used for recording event information occurring at runtime. Generally, the log mainly includes time, level, class name, thread name and main body content of the log, records events occurring during system running, generated data, program running timestamp and context information. As an important record of device running state, the log is an important source for analyzing the running state of the field device. In order to balance real-time and data comprehensiveness, most devices currently adopt the real-time log plus batch log scheme. Therefore, the processing mode of massive log real-time log and batch log is very important.

[0059] Most of the existing modes are based on message queues, and the real-time log data and batch log data generated during device running are all imported into the message queue and processed by the batch flow integrated processing framework. However, whether adopting real-time log data and batch log data closed processing or flow batch integrated processing, the full amount of real-time log data and the full amount of batch log data need to be processed, and the full amount of batch log contains a large amount of repeated data relative to the real-time log, so the full amount processing mode wastes a large amount of computing resources and causes processing pressure to the subsequent process.

[0060] To solve the above technical problems, the embodiment of the present specification provides a data fusion processing mode. According to the time sequence characteristics of log data, the length of time period is set according to different log processing requirements, and the data in each time period can be processed respectively, so that the log data processing process is more flexible. By removing the repeated data and then fusing the data, the repeated data in the real-time log and the batch log is removed, the calculation process is saved, and the efficiency of log data processing is improved. Through the grouping and mapping process of log data, the comparison rate of data is improved.

[0061] In order for those skilled in the art to better understand the technical solutions in the present specification, the technical solutions in the embodiments of the present specification will be described clearly and completely in conjunction with the drawings in the embodiments of the present specification. Obviously, the described embodiments are only part of the embodiments of the present specification, not all. Based on the embodiments of the present specification, all other embodiments obtained by those skilled in the art without creative labor shall fall within the scope of protection of the present specification.

[0062] As shown in the embodiment of the present specification, a flowchart of a log data fusion processing method is provided. Figure 1

[0063] ​In the embodiments of the present specification, the execution unit or server for performing the log data fusion processing is executed by Figure 1 It can be seen that the method comprises the following steps:

[0064] S101: According to the log processing requirement of the power transmission equipment system, the log processing interval is determined, and the log processing interval is divided to obtain a plurality of preset time periods.

[0065] Different users may have different log processing requirements for the power transmission equipment system, for example: daily settlement of equipment operation data, that is, obtaining the equipment operation data of the day and the operation data of the day before yesterday for log data processing; or monthly settlement of equipment operation data. According to the time sequence characteristics of the log data, the log data can be sorted by time, so after determining the processing interval required by the power transmission equipment system, according to the amount of data in the log processing interval, the log processing interval can be time-sliced to divide into a plurality of preset time periods, so as to analyze the log data in each preset time period.

[0066] S102: Obtain real-time logs generated by the power transmission equipment system in a preset time period, and determine batch logs corresponding to the real-time logs in the power transmission equipment system; wherein the real-time logs are used to record real-time operation data of each device component in the power transmission equipment system, and the batch logs are used to record historical operation data to be processed in each device component in the power transmission equipment;

[0067] Firstly, the full amount of logs containing real-time logs to be processed and batch logs to be processed in the power transmission equipment system are obtained. It should be noted that the data in the real-time logs to be processed is the stream data generated in real time in the power transmission equipment system, and the data in the batch logs to be processed is the batch data generated in the historical operation process of the power transmission equipment system. Because the logs contain time information, according to the preset time periods determined in step S101, the data corresponding to the preset time period in the real-time logs to be processed can be obtained as real-time logs. According to the time stamp corresponding to the preset time period, the batch data in the batch logs to be processed corresponding to the time period of the real-time logs and in the same preset time period are divided as batch logs.

[0068] S103: Filtering the repeated data in the real-time logs and the batch logs, determining the missing data of each preset time period of the power transmission equipment system.

[0069] Based on the performance and data consistency, when the power transmission equipment system is processed, the repeated data is filtered in the front end, and the missing data in each preset time period is determined, so as to reduce the processing pressure of the subsequent process and improve the processing efficiency of the log data. In one or more embodiments of the specification, the repeated data in the real-time log and the batch log in the preset time period is filtered, and the missing data in each preset time period in the power transmission equipment system is determined, which specifically includes:

[0070] Firstly, according to the characteristics of each data in the real-time log or the batch log, such as: the equipment running time, the log collection time, the equipment ID, the log type, the log level and the like contained in the log data. The characteristics of each data form the index field of the data. For example: different characteristics of each data are recorded as field k, when the characteristics of the data include: equipment running time, equipment ID, log type, the equipment running time is recorded as k1, the equipment ID is recorded as k2, and the log type is recorded as k3. Then, each characteristic of the data can be combined to form a unique index field K, wherein K=(k1, k2, k3).

[0071] If the index field is determined, each log data is compared according to the index field, and it is determined whether there is repeated data, which will waste a lot of log processing time and cause time pressure for the subsequent processing process. Therefore, after the index field of the log data is determined, the index field is mapped to the corresponding first data set. It should be noted that: after the data in the real-time log is mapped to the first mapping data set, the data formed is the first mapping flow data; after the data in the batch log is mapped to the first mapping data set, the data formed is the first mapping batch data. The mapping method is not limited in the specification, and the hash mapping is taken as an example: the mapping process of the index field K is L=Hash(K). Through the mapping process, the data in the real-time log and the batch log can be mapped to the corresponding first mapping data set, avoiding the tedious process of comparing each index data, and obtaining higher comparison efficiency.

[0072] In the scenario of massive log data generated by power transmission equipment system, the efficiency of the method of mapping only on the index field, comparing based on the first mapping data set, and obtaining repeated data is relatively low. Therefore, in order to further improve the comparison efficiency and avoid comparing each first mapping data set, the first mapping data set can be further processed. First, according to the preset array, the data in the first mapping data set is subjected to modulo operation to obtain a plurality of second mapping data sets, so as to realize grouping of the first mapping data set. For example, if the preset array is N, the first mapping data set needs to be divided into N groups, then according to the above example of obtaining the first mapping data set in the form of hash mapping, L is subjected to modulo operation and M=L%N to obtain the modulus M. Wherein, the preset group number N is a positive integer, when the value of N is larger, the collision rate of the second mapping data set M is lower, the comparison efficiency is higher, but the corresponding mapping relationship between L and M increases and the occupied space becomes larger. Therefore, the comparison efficiency and the occupied space need to be considered to obtain a suitable preset group number for modulo operation.

[0073] After grouping, a plurality of second mapping data sets are obtained, and the mapping relationship between the first mapping data set and the second mapping data set is determined. In order to facilitate the final comparison, the mapping relationship between the first mapping data set and the second mapping data set is cached, and the mapping relationship is stored in the preset mapping set in the form of programming language. According to the above example: the mapping relationship from L to M is cached, because F is generally hash mapping in programming language, such as HashMap in Java. It has efficient get operation, given Key value m, its value L(m) can be obtained in constant time complexity, so the mapping relationship from L to M can be cached in set F, wherein F=(M, L(m)).

[0074] After the mapping and division of the first mapping data set and the second mapping set are completed, repeated data or missing data are obtained by comparison. The comparison can be divided into comparison of the second mapping data set and comparison of the first mapping data set.

[0075] Because there are multiple cases of data in the first mapping data set being mapped to the same data in the second mapping data set, if there is second mapping batch data in the second mapping data set that is not equal to the second mapping stream data in the second mapping data set, it indicates that the data corresponding to the second mapping batch data in the batch log is missing in the real-time log. Based on the mapping relationship between the first mapping data set and the second mapping data set, the first mapping batch data corresponding to the second mapping batch data in the first mapping data set can be determined. Then the batch data corresponding to the first mapping batch data is obtained from the batch log to obtain the first missing data. Therefore, when there is second mapping batch data that is not equal to the second mapping stream data in the second mapping data set, multiple first missing data of the power transmission equipment system can be filtered out at one time based on the second mapping batch data.

[0076] If the second mapping batch data in the second mapping data set is equal to the second mapping stream data, further comparison of the first mapping data set is required. According to the first mapping data set and the second mapping data set, the second missing data of the power transmission equipment system in each preset time period is obtained, and the process specifically includes:

[0077] First, the second mapping data set is screened to obtain a first subset containing only the second mapping stream data and a second subset excluding the first subset. It can be understood that the second mapping data set is a grouping of the first mapping data set, so there are cases where only stream data is included in a group. Assuming that the real-time log in the preset time period is a set P, and the batch log set is Q. Then the first mapping data set of the real-time log is L P , and the second mapping data set is M P . The first mapping data set of the batch log is L Q , and the second mapping data set is M Q . The process of deleting the first subset to obtain the second subset is: M1=M p -M q , where M1 is the second subset, M q is the first subset. And according to the mapping relationship between the first mapping data set and the second mapping data set, the third subset L1 corresponding to the second subset can be determined.

[0078] After screening out the first subset, according to the mapping relationship between the first mapping data set and the second mapping data set, the third subset in the first mapping data set corresponding to the second subset is obtained. Thus, the second missing data of the power transmission equipment system in each preset time period is obtained according to the second mapping data set and the second subset. Wherein, the second missing data of the power transmission equipment system in each preset time period is obtained according to the second mapping data set and the second subset, specifically including:

[0079] Firstly, the missing batch data in the second mapping set is determined according to the difference set of the second mapping data set and the second subset, and then the corresponding first mapping batch data and the corresponding first mapping flow data of the missing data in the first mapping data set are determined according to the mapping relationship between the first mapping data set and the second mapping data set. The corresponding first mapping batch data and the corresponding first mapping flow data are combined to obtain the fourth subset. The second missing data of the power equipment system in the preset time period is obtained by merging the third subset and the fourth subset. According to the mapping relationship between the first mapping data set and the second mapping data set, the third subset corresponding to the second subset M1 can be obtained. The fourth subset is the set of the first mapping data corresponding to the missing batch data in the second mapping set determined by the difference set of the second mapping data set and the second subset, so the missing batch data M2 = M Q M1, and the fourth subset is L2 = L2 = L p M2 L q M2 The third subset and the fourth subset are taken as the second missing data.

[0080] The first missing data and the second missing data are fused to obtain the missing data of the power equipment system in each preset time period, realize the fusion processing of the log data in multiple scenes, realize the analysis of the first deduplication and then the fusion, and concentrate on analyzing the missing data, and improve the processing efficiency of the log data.

[0081] In one or more embodiments of the present specification, the repeated data in the real-time log and the batch log in the preset time period is filtered, and the missing data of each preset time period in the power equipment system is determined, which specifically includes:

[0082] The data in the real-time log and the data in the batch log in the preset time period are compared piece by piece, and if it is determined that a piece of data exists in both the real-time log and the batch log, the data is repeated and filtered in the front end to avoid duplication. If it is determined that a piece of data exists in the batch log but not in the real-time log, or exists in the real-time log but not in the batch log, it is determined that the data is the missing data of the preset time period in the power equipment system.

[0083] S104: Based on the missing data of each preset time period, the abnormal situation in the power equipment system is determined to complete the fusion processing of the log data.

[0084] In one or more embodiments of the present specification, based on the missing data of each preset time period, an abnormal situation in the power transmission equipment system is determined, specifically including the following steps: first, according to the pre-stored data anomaly list, the abnormal reason corresponding to each data in the missing data is obtained. At the same time, according to the characteristics of each data in the missing data, the equipment ID corresponding to each data is determined, and the model of the equipment is determined according to the equipment ID. The determined abnormal reason, preset time period and equipment model are input into the pre-trained deep learning network to obtain the equipment parameter adjustment strategy, so as to return the equipment parameter adjustment strategy to the corresponding equipment according to the equipment ID to adjust the running state of the equipment, so as to ensure the safe operation of the power transmission equipment system.

[0085] As shown in Figure 2 , one or more embodiments of the present specification provide an internal structure diagram of a log data fusion processing device.

[0086] As can be seen from Figure 2 , a log data fusion processing device includes:

[0087] at least one processor 201, and

[0088] a memory 202 in communication connection with the at least one processor 201, wherein

[0089] The memory 202 stores executable instructions of the at least one processor 201, so that the at least one processor 201 can:

[0090] According to the log processing requirement of the power transmission equipment system, a log processing interval is determined, and the log processing interval is divided to obtain a plurality of preset time periods;

[0091] Obtain real-time logs generated by the power transmission equipment system in a preset time period, and determine batch logs corresponding to the real-time logs in the power transmission equipment system;

[0092] Filtering repeated data in the real-time logs and the batch logs, determining missing data of each preset time period of the power transmission equipment system;

[0093] Based on the missing data of each preset time period, the abnormal situation in the power transmission equipment system is determined to complete the fusion processing of the log data.

[0094] Each embodiment in the present specification is described in a progressive manner, and the same and similar parts between each embodiment can be referred to each other. Each embodiment focuses on the difference from other embodiments. In particular, for the device, equipment and non-volatile computer storage medium embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the part of the method embodiment.

[0095] The above-described embodiments of the application have several aspects, no single one of which is solely responsible for the application's desirable attributes. Without limiting the scope of this application, other aspects of the application will become apparent from consideration of the drawings and following detailed description, it being understood that such changes in the state of the art can be made without departing from the spirit and scope of the application.

[0096] The foregoing description of one or more implementations will be better understood in view of the accompanying drawings in which: The above description is intended to be illustrative and not restrictive. Many other implementations will be apparent to those of skill in the art upon reading and understanding the above description. The scope of the application should, therefore, be determined with reference to the appended claims, along with the full scope of equivalents to which such claims are entitled.

Claims

1. A log data fusion processing method characterized by comprising: The method comprises: According to the log processing requirements of the power transmission equipment system, determine the log processing interval; and divide the log processing interval to obtain a plurality of preset time periods; Obtain the real-time log generated by the power transmission equipment system in the preset time period, and determine the batch log corresponding to the real-time log in the power transmission equipment system; wherein the real-time log is used to record the real-time running data of each device component in the power transmission equipment system, and the batch log is used to record the historical running data to be processed in each device component in the power transmission equipment; Filter the repeated data in the real-time log and the batch log, and determine the missing data of each preset time period of the power transmission equipment system; Based on the missing data of each preset time period, determine the abnormal situation in the power transmission equipment system, so as to complete the fusion processing of the log data according to the missing data; The filtering of the repeated data in the real-time log and the batch log, and the determination of the missing data of each preset time period of the power transmission equipment system, specifically comprises: Performing a preset first processing on the data in the real-time log and the batch log to obtain a first mapping data set; Group processing the data in the first mapping data set to obtain a plurality of second mapping data sets, and determining the mapping relationship between the first mapping data set and the second mapping data set; wherein the data in the second mapping data set includes one or more of second mapping batch data and second mapping flow data; If it is determined that the second mapping batch data is not equal to the second mapping flow data, the first missing data of each preset time period of the power transmission equipment system is determined according to the second mapping batch data; If it is determined that the second mapping batch data is equal to the second mapping flow data, the second missing data of each preset time period of the power transmission equipment system is obtained according to the first mapping data set and the second mapping data set; Fuse the first missing data and the second missing data to obtain the missing data of each preset time period of the power transmission equipment system.

2. The log data fusion processing method of claim 1, wherein, The obtaining of the real-time log generated by the power transmission equipment system in the preset time period, and the determination of the batch log corresponding to the real-time log in the power transmission equipment system, specifically comprises: Obtain the full log of the power transmission equipment system, wherein the full log includes the to-be-processed real-time log and the to-be-processed batch log; Determine the data of the to-be-processed real-time log in the preset time period as the real-time log; According to the time stamp corresponding to the preset time period, divide the data corresponding to the time period of the real-time log in the to-be-processed batch log as the batch log.

3. The log data fusion processing method of claim 1, wherein, The first mapping data set is obtained by performing a preset first processing on the data in the real-time log and the batch log, specifically comprising: According to the characteristics of the data in the real-time log and the batch log, determine the index field of the data; wherein the characteristics include any one or more of the following: device running time, log collection time, device ID, log type; The index field is mapped to a corresponding first mapping data set; wherein the data in the first mapping data set includes first mapping batch data and first mapping stream data.

4. The log data fusion processing method of claim 1, wherein, The data in the first mapping data set is grouped to obtain a plurality of second mapping data sets, and a mapping relationship between the first mapping data set and the second mapping data set is determined, specifically including: According to a preset array, the data in the first mapping data set is subjected to a modulo operation to obtain a plurality of second mapping data sets; According to the modulo operation process, the mapping relationship between the first mapping data set and the second mapping data set is determined, and the mapping relationship is stored in a preset mapping set in the form of a programming language.

5. The log data fusion processing method of claim 1, wherein, If it is determined that the second mapping batch data is not equal to the second mapping stream data, the first missing data of each preset time period of the power transmission equipment system is determined according to the second mapping batch data, specifically including: If the second mapping batch data is not in the second mapping stream data, the corresponding first mapping batch data of the second mapping batch data in the first mapping data set is determined based on the mapping relationship between the first mapping data set and the second mapping data set; The batch data corresponding to the first mapping batch data is obtained in the batch log, and the batch data is taken as the first missing data.

6. The log data fusion processing method of claim 1, wherein, If it is determined that the second mapping batch data is equal to the second mapping stream data, the second missing data of each preset time period of the power transmission equipment system is obtained according to the first mapping data set and the second mapping data set, specifically including: The plurality of second mapping data sets are screened to determine a first subset of the second mapping data sets, and a second subset of the second mapping data sets is obtained; wherein the first subset only includes second mapping stream data; According to the mapping relationship between the first mapping data set and the second mapping data set, a third subset corresponding to the second subset is obtained; wherein the third subset is located in the first mapping data set; According to the plurality of second mapping data sets and the second subset, the second missing data of each preset time period of the power transmission equipment system is obtained.

7. The log data fusion processing method of claim 6, wherein, According to the second mapping data set and the second subset, the second missing data of each preset time period of the power transmission equipment system is obtained, specifically including: According to the difference set of the second mapping data set and the second subset, the missing batch data in the second mapping set is determined; According to the mapping relationship, the corresponding first mapping batch data and the corresponding first mapping stream data of the missing batch data in the first mapping data set are determined; and according to the corresponding first mapping batch data and the corresponding first mapping stream data, a fourth subset is obtained; The data in the third subset and the fourth subset are merged to obtain the second missing data of each preset time period of the power transmission equipment system.

8. The log data fusion processing method of claim 1, wherein, The abnormal situation in the power transmission equipment system is determined based on the missing data of each preset time period, specifically including: According to the pre-stored data exception list, the abnormal reason corresponding to each data in the missing data is obtained; According to the characteristics of the data, the device ID corresponding to each data in the missing data is determined, and the device model is determined based on the device ID; The abnormal reason, the preset time period and the device model are input into a pre-trained deep learning network to output a device parameter adjustment strategy; Based on the device ID, the device parameter adjustment strategy is traced back to the device to adjust the running state of the device.

9. A log data fusion processing device characterized by comprising: The device comprises: at least one processor, and a memory connected in communication with the at least one processor, wherein the memory stores executable instructions of the at least one processor, so that the at least one processor can: determine a log processing interval according to the log processing requirement of the power transmission equipment system; and divide the log processing interval to obtain a plurality of preset time periods; obtain real-time logs generated by the power transmission equipment system in a preset time period, and determine batch logs corresponding to the real-time logs in the power transmission equipment system; filtering repeated data in the real-time logs and the batch logs, determining missing data of each preset time period of the power transmission equipment system; based on the missing data of each preset time period, determining the abnormal situation in the power transmission equipment system to complete the fusion processing of the log data; the filtering of repeated data in the real-time logs and the batch logs, determining the missing data of each preset time period of the power transmission equipment system, specifically includes: performing a preset first processing on the data in the real-time logs and the batch logs to obtain a first mapping data set; grouping the data in the first mapping data set to obtain a plurality of second mapping data sets, and determining the mapping relationship between the first mapping data set and the second mapping data set; wherein the data in the second mapping data set includes one or more of second mapping batch data and second mapping stream data; if it is determined that the second mapping batch data is not equal to the second mapping stream data, the first missing data of each preset time period of the power transmission equipment system is determined according to the second mapping batch data; if it is determined that the second mapping batch data is equal to the second mapping stream data, the second missing data of each preset time period of the power transmission equipment system is obtained according to the first mapping data set and the second mapping data set; fuse the first missing data and the second missing data to obtain the missing data of each preset time period of the power transmission equipment system.

Images