Data processing method, processor, apparatus, and storage medium

Abstract

The application relates to the computer technical field, in particular to a data processing method, a processor, an apparatus and a storage medium. The method comprises the following steps: acquiring interface monitoring rules configured for each data interface, and saving the interface monitoring rules in redis; acquiring log data of each data interface in real time, and storing the log data in a message queue; using flink to consume the log data in the message queue in real time to obtain a log data stream of each piece of log data of each data interface; consuming the interface monitoring rules stored in the redis in real time to obtain monitoring rule data streams of each data interface; fusing the log data streams and the monitoring rule data streams of each data interface to obtain fusion data streams of each data interface; and analyzing the fusion data streams according to the monitoring rule data streams contained in the fusion data streams to obtain monitoring analysis data of each data interface. Through real-time analysis of the log data of the data interface, the management efficiency is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer, in particular, to a data processing method, a processor, an apparatus and a storage medium. BACKGROUND

[0002] In the field of computer technology, with the improvement of various big data components, most components can provide data query function to the outside, for data access of application layer. In order to effectively control data security and permission, big data management platform will integrate data service platform. Since the data volume of big data interface service is large, there is a high requirement for interface data transmission. In order to ensure service quality and avoid interface timeout, it is necessary to effectively manage and monitor the interfaces managed by the data service platform.

[0003] In the prior art, the ELK (Elasticsearch+Logstash+Kabana) technical solution is mainly used to collect the interface operation log of the data service managed by the data service platform, store it in the index database Elasticsearch, and finally display the log data through Kabana. When the data service has a problem, the operation and maintenance personnel can check and check the log through Kabana to find out the cause of the problem, and then manually maintain and solve the problem. This is a passive way to solve the problem. When the service is unavailable and the data is delayed, it will affect the normal operation of the customer application system. SUMMARY

[0004] The purpose of the present application is to provide a data processing method, a processor, an apparatus and a storage medium for real-time monitoring and analysis of log data.

[0005] In order to achieve the above purpose, the present application provides a data processing method applied to a data service platform, the data processing method comprising:

[0006] Obtaining the interface monitoring rules configured for each data interface, and saving the interface monitoring rules to redis;

[0007] Real-time acquisition of log data of each data interface, and storage of the log data to a message queue;

[0008] Real-time consumption of the log data in the message queue by flink to obtain the log data stream of each log data of each data interface;

[0009] Real-time consumption of the interface monitoring rules stored in redis to obtain the monitoring rule data stream of each data interface;

[0010] Fusion of the log data stream and the monitoring rule data stream of each data interface to obtain the fusion data stream of each data interface;

[0011] The fusion data stream is analyzed according to the monitoring rule data stream contained in the fusion data stream to obtain monitoring analysis data of each data interface.

[0012] In the embodiments of the present application, the interface monitoring rule includes a monitoring time period, and the data processing method further includes: after obtaining the interface monitoring rule configured for each data interface, determining the greatest common divisor of the monitoring time periods of all data interfaces according to the monitoring time period of each data interface; determining the greatest common divisor as the detection time length of the detection window of the log data; wherein, when analyzing the log data stream in the fusion data stream through the detection window, the log data stream is divided into a plurality of sub-log data streams according to the detection time length, and log detection results of each sub-log data stream are obtained.

[0013] In the embodiments of the present application, the real-time consumption of the log data in the message queue by flink to obtain the log data stream of each log data of each data interface includes: detecting the log data of each data interface to determine whether the real log data is missing in the detection time length for any one data interface; in the case of real log data missing, a preset number of simulated log data are generated to supplement the log data of the data interface missing the real log data, so as to obtain the log data stream of each log data of each data interface.

[0014] In the embodiments of the present application, determining the analysis rule for the fusion data stream according to the monitoring rule data stream contained in the fusion data stream to obtain the monitoring analysis data of each data interface includes: transmitting the fusion data stream to the detection window to determine the data transmission success rate of each sub-log data stream, and storing the data transmission success rate to the ring queue; determining the monitoring time period of each data interface according to the monitoring rule data stream contained in the fusion data stream; for each data interface, determining the first number of detection time lengths required to reach the monitoring time period of the data interface; for each data interface, selecting the data transmission success rate of the first number of sub-log data streams corresponding to the data interface from the ring queue according to the chronological order; for each data interface, determining the monitoring analysis data of the data interface according to the data transmission success rate of the first number of sub-log data streams.

[0015] In the embodiments of the present application, the interface monitoring rule includes a success rate range in each monitoring time period, and the monitoring analysis data of the data interface is determined according to the data transmission success rate of the first number of sub-log data streams, including: determining the average value of the data transmission success rate of the first number of sub-log data streams; in the case that the average value is in the success rate range, determining that the log data stream corresponding to the data interface meets the interface monitoring rule of the data interface; in the case that the average value is not in the success rate range, determining that the log data stream corresponding to the data interface does not meet the interface monitoring rule of the data interface.

[0016] In the embodiments of the present application, the data processing method further includes: in the case that it is determined that the log data stream corresponding to the data interface meets the interface monitoring rule of the data interface, obtaining feedback information for the data interface; in the case that the feedback information is transmission abnormal, determining that the interface monitoring rule needs to be modified; in the case that the feedback information is transmission normal, determining that the data transmission of the data interface is normal.

[0017] In the embodiments of the present application, the real-time consumption of the interface monitoring rule stored in the redis to obtain the monitoring rule data stream of each data interface includes: after determining that the interface monitoring rule needs to be modified, in the case that the updated interface monitoring rule is obtained, the updated interface monitoring rule is broadcasted in the flink cluster in real time to cover the interface monitoring rule before the update.

[0018] The second aspect of the present application provides a processor configured to execute the data processing method of any one of the above.

[0019] The third aspect of the present application provides a data processing device, including the above processor.

[0020] The fourth aspect of the present application provides a machine readable storage medium, the machine readable storage medium stores instructions, the instructions make the processor configured to execute the data processing method according to any one of the above when the processor executes the instructions.

[0021] Through the above technical solution, the log data of the data interface can be obtained in real time, the flink is used for fusion analysis of the log data and the interface monitoring rule of the data interface, the monitoring analysis data of each data interface is obtained in real time, the transmission of the service interface of the massive big data is monitored in real time, and the problems generated in the transmission process are investigated in time, thereby improving the management efficiency.

[0022] Other features and advantages of the present application will be described in detail in the following specific embodiments. BRIEF DESCRIPTION OF DRAWINGS

[0023] The accompanying drawings are included to provide a further understanding of the application and are incorporated in and constitute a part of this specification, illustrate embodiments of the application and serve to explain the principles of the application, but are not intended to limit the application. In the drawings:

[0024] Figure 1 A flowchart of a data processing method according to an embodiment of the application is schematically shown;

[0025] Figure 2 An internal structure diagram of a computer device according to an embodiment of the application is schematically shown. DETAILED DESCRIPTION

[0026] The specific embodiments of the application will be described below in detail with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are merely intended to explain and illustrate the application, and are not intended to limit the application.

[0027] It should be noted that if the embodiments of the application involve directional indications (such as up, down, left, right, front, back, etc.), the directional indications are only used to explain the relative positional relationship, movement, etc. between the components in a certain posture (as shown in the drawings), and if the certain posture changes, the directional indications will also change accordingly.

[0028] In addition, if the embodiments of the application involve descriptions of "first", "second", etc., the descriptions of "first", "second", etc. are only for description purposes, and cannot be understood as indicating or implying the relative importance of the indicated technical features or implicitly indicating the number of the indicated technical features. Therefore, the features limited by "first" and "second" can explicitly or implicitly include at least one of the features. In addition, the technical solutions of the various embodiments can be combined with each other, but it must be based on the fact that a person skilled in the art can realize it, and when the combination of technical solutions appears to be contradictory or unachievable, it should be considered that the combination of technical solutions does not exist and is not within the scope of protection claimed by the application.

[0029] Figure 1 A flowchart of a data processing method according to an embodiment of the application is schematically shown. As shown in Figure 1 In one embodiment of the application, a data processing method is provided, comprising the following steps:

[0030] Step 101, obtaining an interface monitoring rule configured for each data interface, and saving the interface monitoring rule to redis;

[0031] Step 102, obtaining log data of each data interface in real time, and storing the log data to a message queue;

[0032] Step 103, real-time flink is used to consume log data in the message queue to obtain the log data stream of each data interface;

[0033] Step 104, real-time consumption of the interface monitoring rules stored in redis to obtain the monitoring rule data stream of each data interface;

[0034] Step 105, fusion of the log data stream and the monitoring rule data stream of each data interface to obtain the fusion data stream of each data interface;

[0035] Step 106, according to the monitoring rule data stream contained in the fusion data stream, the fusion data stream is analyzed to obtain the monitoring analysis data of each data interface.

[0036] The data service platform is provided with a plurality of data interfaces for transmitting data, and the processor can obtain the interface monitoring rules configured for each data interface in the data service platform, and save the obtained interface monitoring rules to redis. The processor can obtain the log data of each data interface in real time, and store the log data in the message queue. The log data of the data interface can include various behavior data generated in the process of interaction of the data interface, such as project name, service module name, source IP address, caller name, interface address, port number, call success flag, interface call start time, call time, message content, log level, etc. The processor can collect the above log data in real time.

[0037] After the processor stores the obtained log data in the message queue, it can use the flink computing engine to consume the log data stored in the message queue in real time, thereby obtaining the log data stream corresponding to each log data of each data interface. The processor can consume the interface monitoring rules stored in redis in real time, thereby obtaining the monitoring rule data stream corresponding to each data interface.

[0038] After the processor obtains the log data stream and the monitoring rule data stream of each data interface, it can fuse the log data stream and the monitoring rule data stream of each data interface to obtain the fusion data stream of each data interface. According to the monitoring rule data stream contained in the obtained fusion data stream of each data interface, the fusion data stream is analyzed to obtain the monitoring analysis data of each data interface.

[0039] In an embodiment, the interface monitoring rule comprises a monitoring time period, and the data processing method further comprises: after obtaining the interface monitoring rule configured for each data interface, determining a greatest common divisor of the monitoring time periods of all data interfaces according to the monitoring time period of each data interface; determining the greatest common divisor as a detection time length of a detection window for detecting the log data; and when analyzing the log data stream in the fused data stream through the detection window, dividing the log data stream into a plurality of sub-log data streams according to the detection time length, and obtaining a log detection result of each sub-log data stream.

[0040] The interface monitoring rule of the data interface configuration can comprise a monitoring time period of the data interface. After obtaining the interface monitoring rule configured for each data interface, the processor can determine a greatest common divisor of the monitoring time periods of all data interfaces according to the monitoring time period of each data interface. The processor can determine the greatest common divisor as a detection time length of a detection window for detecting and analyzing the log data. For example, assuming that the monitoring time period of the A data interface is 6 minutes, the monitoring time period of the B data interface is 9 minutes, and the monitoring time period of the C data interface is 15 minutes, the processor determines that the greatest common divisor of the monitoring time periods of the A, B, and C data interfaces is 3. Then, the processor can determine the detection time length of the detection window as 3 minutes.

[0041] After the processor determines the detection time length of the detection window, when analyzing the log data stream in the fused data stream through the detection window, the processor can divide the log data stream into a plurality of sub-log data streams according to the detection time length, and obtain a log detection result of each sub-log data stream. For example, assuming that the processor sets the detection time length of the detection window as 3 minutes, when analyzing the log data stream in the fused data stream through the detection window, the processor can divide the log data stream into a plurality of sub-log data streams according to the time length of 3 minutes, that is, the length of each sub-log data stream is 3 minutes, and obtain a log detection result of each sub-log data stream through the detection window.

[0042] In an embodiment, the real-time consumption of the log data in the message queue by the flink to obtain the log data stream of each log data of each data interface comprises: detecting the log data of each data interface to determine whether a real log data missing occurs in any one data interface within the detection time length; and in the case of a real log data missing, generating a preset number of simulated log data to supplement the log data of the data interface missing the real log data, thereby obtaining the log data stream of each log data of each data interface.

[0043] The processor can detect the log data of each data interface through the flink computing engine to determine whether real log data is missing for any data interface in a detection time length, wherein the real log data refers to real behavior data generated by the data interface in a data interaction process. Since the calling of the data interface does not occur every moment, the real log data is not generated all the time, and thus the real log data may be missing. When it is determined that the real log data is missing, the flink computing engine can generate a preset number of simulated log data to supplement the log data of the data interface for which the real log data is missing, so that each data interface has log data in any detection time length, thereby obtaining a log data stream of each log data of each data interface. For example, it is assumed that the A data interface does not generate log data between 10:00 and 10:09, and it is assumed that the detection time length of the detection window is 3 minutes. Therefore, the A data interface does not have real log data in 3 detection time lengths. In order to ensure that the detection window can be triggered in real time each time, the processor can supplement the log data in any detection time length for which the real log data is missing through the flink computing engine. The processor can set the preset number to 1, that is, the flink computing engine can generate one simulated log data for any detection time length for which the real log data is missing to supplement the log data of the data interface. For example, the flink computing engine can supplement 3 simulated log data for 10:00 to 10:09 to obtain the log data stream of the A data interface.

[0044] In one embodiment, determining the analysis rule for the fusion data stream according to the monitoring rule data stream included in the fusion data stream to obtain the monitoring analysis data of each data interface comprises: transmitting the fusion data stream to a detection window to determine the data transmission success rate of each sub-log data stream, and storing the data transmission success rate to a ring queue; determining the monitoring time period of each data interface according to the monitoring rule data stream included in the fusion data stream; determining, for each data interface, a first number of detection time lengths required to reach the monitoring time period of the data interface; for each data interface, selecting the data transmission success rate of the first number of sub-log data streams corresponding to the data interface from the ring queue in chronological order; and for each data interface, determining the monitoring analysis data of the data interface according to the data transmission success rate of the first number of sub-log data streams.

[0045] The processor can transmit the fusion data stream to a detection window set up by the processor when determining the analysis rule for the fusion data stream according to the monitoring rule data stream contained in the fusion data stream to obtain the monitoring analysis data of each data interface, and determine the data transmission success rate of each sub-log data stream through the detection window and store the data transmission success rate of each sub-log data stream in the ring queue. The processor can determine the monitoring time period of each data interface according to the monitoring rule data stream contained in the fusion data stream, and determine the first number of detection time lengths required to reach the monitoring time period of each data interface. For each data interface, the processor can select the data transmission success rate of the first number of sub-log data streams of the data interface from the ring queue according to the determined first number, and determine the monitoring analysis data of each data interface according to the data transmission success rate of the first number of sub-log data streams.

[0046] For example, after the fusion data stream is transmitted to the detection window, the detection window can determine the data transmission success rate of each sub-log data stream, and store the data transmission success rate of all sub-log data streams of all data interfaces in the ring queue. Assuming that the processor determines that the monitoring time period of the A data interface is 6 minutes, the monitoring time period of the B data interface is 9 minutes, and the monitoring time period of the C data interface is 15 minutes according to the monitoring data stream contained in the fusion data stream, and assuming that the detection time length of the detection window is 3 minutes, the processor can determine that 2 detection time lengths are required to reach the A data interface, 3 detection time lengths are required to reach the B data interface, and 5 detection time lengths are required to reach the C data interface. After determining the first number of detection time lengths required to reach the monitoring time period of each data interface, the processor can select the success rate of 2 sub-log data streams of the A data interface from the ring queue in chronological order to determine the monitoring analysis data of the A data interface, the processor can select the success rate of 3 sub-log data streams of the B data interface from the ring queue in chronological order to determine the monitoring analysis data of the B data interface, and the processor can select the success rate of 5 sub-log data streams of the C data interface from the ring queue to determine the monitoring analysis data of the C data interface.

[0047] In one embodiment, the interface monitoring rule includes a success rate range in each monitoring time period, and determining the monitoring analysis data of each data interface according to the data transmission success rate of the first number of sub-log data streams includes: determining the average value of the data transmission success rate of the first number of sub-log data streams; in the case that the average value is within the success rate range, determining that the log data stream corresponding to the data interface meets the interface monitoring rule of the data interface; and in the case that the average value is not within the success rate range, determining that the log data stream corresponding to the data interface does not meet the interface monitoring rule of the data interface.

[0048] The interface monitoring rule of the data interface configuration includes a success rate range in each monitoring time period. After the processor selects the first number of sub-log data streams for each data interface, the processor can obtain the transmission success rate of each sub-log data stream and determine the average value of the transmission success rates of the first number of sub-log data streams. If the average value is within the success rate range set by the interface monitoring rule, the processor can determine that the log data stream corresponding to the data interface meets the interface monitoring rule of the data interface. If the average value is not within the success rate range set by the interface monitoring rule, the processor can determine that the log data stream corresponding to the data interface does not meet the interface monitoring rule of the data interface. At this time, the processor can perform an alarm prompt and push an alarm message to an operation and maintenance personnel. For example, assuming that the processor obtains the transmission success rates of 5 sub-log data streams for the C data interface in the ring queue, the processor can determine the average transmission success rate of the 5 sub-log data streams and compare the obtained average transmission success rate with the success rate range set by the interface monitoring rule of the C data interface. If the average transmission success rate is within the success rate range, it is determined that the log data stream corresponding to the C data interface meets the interface monitoring rule of the C data interface. If the average transmission success rate is not within the success rate range, it is determined that the log data stream corresponding to the C data interface does not meet the interface monitoring rule of the C data interface. The processor can perform an alarm prompt for the C data interface.

[0049] In one embodiment, in a case where it is determined that the log data stream corresponding to the data interface meets the interface monitoring rule of the data interface, feedback information for the data interface is obtained. In a case where the feedback information indicates transmission abnormality, it is determined that the interface monitoring rule needs to be modified. In a case where the feedback information indicates normal transmission, it is determined that the data transmission of the data interface is normal.

[0050] In a case where it is determined that the log data stream corresponding to the data interface meets the interface monitoring rule of the data interface, the processor can obtain feedback information for the data interface, which can be input by a user. When the feedback information indicates data transmission abnormality, it indicates that a failure occurs when the user uses the data interface, but the interface monitoring rule configured at present does not detect data monitoring abnormality, so the interface monitoring rule configured may not be suitable for the data interface. Therefore, in a case where the feedback information indicates transmission abnormality, the processor can determine that the interface monitoring rule needs to be modified. If it is determined that the log data stream corresponding to the data interface meets the interface monitoring rule of the data interface and the feedback information indicates normal transmission, the processor can determine that the data transmission of the data interface is normal.

[0051] In one embodiment, the real-time consumption of the interface monitoring rule stored in the redis includes: determining that the interface monitoring rule is to be modified, and in a case where the updated interface monitoring rule is obtained, performing real-time broadcasting of the updated interface monitoring rule in the flink cluster to overwrite the interface monitoring rule before the update.

[0052] In a case where the processor determines that the interface monitoring rule of the data interface is to be modified, the processor can perform real-time broadcasting of the updated interface monitoring rule in the flink cluster to overwrite the interface monitoring rule before the update.

[0053] In one embodiment, a processor configured to perform any of the above data processing methods is provided.

[0054] The data service platform is provided with a plurality of data interfaces for transmitting data. The processor can obtain the interface monitoring rule configured for each data interface in the data service platform and save the obtained interface monitoring rule to the redis. The processor can also obtain the log data of each data interface in real time, wherein the log data can include various behavior data generated in the process of interaction of the data interface, such as project name, service module name, source IP address, caller name, interface address, port number, call success flag, interface call start time, call duration, message content, log level, etc. The processor can establish a detection window to detect the log data of each data interface through the detection window, thereby analyzing the log data generated by each data interface and determining whether it conforms to the interface monitoring rule of the corresponding data interface.

[0055] After the processor obtains the interface monitoring rule of each data interface, the processor can obtain the monitoring time period for each data interface included in each interface monitoring rule, and determine the greatest common divisor of the monitoring time periods of all data interfaces according to each monitoring time period. The processor can determine the greatest common divisor as the detection time length of the detection window for detecting and analyzing the log data. After the processor determines the detection time length of the detection window, the processor can cut the log data stream into a plurality of sub-log data streams according to the detection time length when analyzing the log data stream in the fusion data stream through the detection window, and obtain the log detection result of each sub-log data stream.

[0056] For example, assuming that the monitoring time period of the A data interface is 6 minutes, the monitoring time period of the B data interface is 9 minutes, and the monitoring time period of the C data interface is 15 minutes, the processor determines that the greatest common divisor of the monitoring time periods of the A, B, and C data interfaces is 3. The processor can then determine the detection time length of the detection window to be 3 minutes. When the detection window is used to analyze the log data stream in the fused data stream, the log data stream can be divided into a plurality of sub-log data streams according to the time length of 3 minutes, i.e., each sub-log data stream has a length of 3 minutes, and the detection window obtains a log detection result for each sub-log data stream.

[0057] The processor acquires log data of each data interface in real time, stores a message queue, and then consumes the log data in the message queue through the flink computing engine. When the flink computing engine consumes the log data, it detects whether a real log data loss occurs within the detection time length. This is because the calling of the data interface does not occur at every moment, so real log data is not always generated, and thus real log loss may occur. When real log data loss occurs within the detection time length, the detection window set cannot be triggered by the log data stream, thereby affecting the detection of subsequent log data streams. Therefore, the flink computing engine generates simulated log data to supplement the log data stream when it determines that log data loss occurs, so as to ensure that the detection window is triggered. The simulated log data supplements the time period of the missing log data, and the generated simulated log data is only used to trigger the detection window and does not affect the detection and analysis of real log data by the detection window. The processor can set a preset number, and the flink computing engine can generate a preset number of simulated log data within the detection time period of the missing log data stream according to the set preset number. For example, assuming that the window time length of the detection window is 3 minutes, and assuming that no log data is generated by the data interface within a continuous 3 minutes, the detection window cannot be triggered when it detects this. The processor can set the preset number to 1, and the flink computing engine can generate 1 simulated log data, thereby ensuring the triggering of the detection window. The flink computing engine supplements the log data of each data interface with simulated log data, thereby obtaining a continuous log data stream of each data interface.

[0058] The processor can consume the interface monitoring rules stored in the redis in real time, thereby obtaining a monitoring rule data stream corresponding to each data interface. The processor can fuse the log data stream of each data interface and the monitoring rule data stream, to obtain a fused data stream of each data interface. The fused data stream is input to a detection window set to be detected. The detection window detects and analyzes each fused data stream according to a detection time length, and obtains a detection result of the log data stream of each detection time length, that is, a log detection result of each sub-log data stream. After obtaining the log detection result of each sub-log data stream of each data interface, the processor can store the log detection result of the sub-log data stream of all data interfaces in a ring queue. The processor can determine, for each data interface, a first number of detection time lengths required to reach a monitoring time period of the data interface of the data interface, and select, in the ring queue, the detection result of the first number of sub-log data streams of the corresponding data interface in the order of time, wherein the detection result can include a transmission success rate of the sub-log data stream. Then, the average value of the transmission success rate of the first number of sub-log data streams is determined, and the average value is compared with a transmission success rate range set by the interface monitoring rule of the data interface. If the average value is within the transmission success rate range set by the successful interface monitoring rule, it is determined that the log data of the data interface meets the interface monitoring rule of the data interface. In the case where the average value is not within the success rate range set by the interface monitoring rule, the processor can determine that the log data stream corresponding to the data interface does not meet the interface monitoring rule of the data interface. At this time, the processor can alarm and prompt, and push an alarm information to an operation and maintenance personnel.

[0059] For example, after the fusion data stream is transmitted to the detection window, the detection window can determine the data transmission success rate of each sub-log data stream, and store the data transmission success rate of all sub-log data streams of the data interface into the ring queue. Assuming that the processor determines that the monitoring time period of the A data interface is 6 minutes, the monitoring time period of the B data interface is 9 minutes, and the monitoring time period of the C data interface is 15 minutes according to the monitoring data stream contained in the fusion data stream, and assuming that the detection time length of the detection window is 3 minutes, the processor can determine that 2 detection time lengths are needed to reach the A data interface, 3 detection time lengths are needed to reach the B data interface, and 5 detection time lengths are needed to reach the C data interface. After determining the first number of detection time lengths needed to reach the monitoring time period of each data interface, the processor can select the success rates of 2 sub-log data streams of the A data interface from the ring queue in chronological order, and determine the average value of the transmission success rates of the 2 sub-log data streams of the A data interface. The average value is compared with the success rate range set by the interface monitoring rule of the A data interface. If the success rate range is within the success rate range, it is determined that the log data stream corresponding to the A data interface meets the interface monitoring rule of the A data interface. If it is not within the success rate range, it is determined that the log data stream corresponding to the A data interface does not meet the interface monitoring rule of the A data interface. The processor can alarm and prompt the A data interface. Similarly, the processor can select the success rates of 3 sub-log data streams of the B data interface from the ring queue in chronological order to determine the monitoring analysis data of the B data interface. The processor can select the success rates of 5 sub-log data streams of the C data interface from the ring queue in chronological order to determine the monitoring analysis data of the C data interface.

[0060] When the processor determines that the log data stream corresponding to the data interface meets the interface monitoring rule of the data interface, the processor can also obtain feedback information for the data interface. The feedback information can be input by the user. When the feedback information indicates that the data transmission is abnormal, it means that the user has encountered a failure when using the data interface. However, the interface monitoring rule configured according to the current data interface does not detect the data monitoring abnormality, so the configured interface monitoring rule may not be suitable for the data interface. Therefore, when the feedback information is transmission abnormal, the processor can determine that the interface monitoring rule needs to be modified, and prompt the operation and maintenance personnel. When the operation and maintenance personnel modify the interface monitoring rule, the processor can broadcast the updated interface monitoring rule in the flink cluster in real time to overwrite the interface monitoring rule before the update. If the processor determines that the log data stream corresponding to the data interface meets the interface monitoring rule of the data interface, and the feedback information indicates that the transmission is normal, the processor can determine that the data transmission of the data interface is normal.

[0061] The technical scheme can acquire log data of a data interface in real time, perform fusion analysis on the log data and interface monitoring rules of the data interface through flink, set the detection time length of a detection window as a common multiple of monitoring periods of all data interfaces, thereby detecting the log data of all data interfaces, and utilize a ring queue to statistically analyze detection data of the data interfaces with different monitoring periods, thereby monitoring transmission of a service interface of massive data in real time, and timely troubleshooting problems generated in the transmission process, thereby improving management efficiency.

[0062] The memory can include non-persistent memory in a computer readable medium, random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash memory (flash RAM), and the memory includes at least one memory chip.

[0063] In one embodiment, a computer device, which can be a server, is provided, and an internal structure diagram of the computer device can be as shown in Figure 2 The computer device includes a processor A01, a network interface A02, a memory (not shown in the figure) and a database (not shown in the figure) connected through a system bus. The processor A01 of the computer device is configured to provide computing and control capabilities. The memory of the computer device includes an internal memory A03 and a non-volatile storage medium A04. The non-volatile storage medium A04 stores an operating system B01, a computer program B02 and a database (not shown in the figure). The internal memory A03 provides an environment for running the operating system B01 and the computer program B02 in the non-volatile storage medium A04. The database of the computer device is configured to store related data of a construction machine and related data input by an operator. The network interface A02 of the computer device is configured to communicate with an external terminal through a network connection. The computer program B02 is executed by the processor A01 to implement a data processing method.

[0064] Figure 1 A flowchart of the data processing method in one embodiment is shown in Figure 1 Although the steps in the flowchart are shown in sequence according to the arrows, the steps are not necessarily executed in sequence according to the arrows. Unless otherwise specified herein, the steps are not strictly limited in sequence, and the steps can be executed in other sequences. Moreover, Figure 1At least one of the steps in the method can comprise a plurality of sub-steps or a plurality of stages, which are not necessarily performed at the same time, but can be performed at different times, and the order of the sub-steps or stages is not necessarily sequential, but can be performed alternately or in rotation with other steps or sub-steps or stages of other steps.

[0065] The embodiment of the present application provides a device, which comprises a processor, a memory, and a program stored in the memory and executable on the processor, and the processor implements the following steps when executing the program: obtaining interface monitoring rules configured for each data interface, and storing the interface monitoring rules in redis; obtaining log data of each data interface in real time, and storing the log data in a message queue; consuming the log data in the message queue in real time by using flink to obtain a log data stream of each piece of log data of each data interface; consuming the interface monitoring rules stored in redis in real time to obtain monitoring rule data streams of each data interface; fusing the log data stream and the monitoring rule data stream of each data interface to obtain a fused data stream of each data interface; and analyzing the fused data stream according to the monitoring rule data stream contained in the fused data stream to obtain monitoring analysis data of each data interface.

[0066] In one embodiment, the interface monitoring rules comprise a monitoring time period, and the data processing method further comprises: after obtaining the interface monitoring rules configured for each data interface, determining a greatest common divisor of monitoring time periods of all data interfaces according to the monitoring time periods of each data interface; determining the greatest common divisor as a detection time length of a detection window of the log data; and when analyzing the log data stream in the fused data stream by using the detection window, the log data stream is divided into a plurality of sub-log data streams according to the detection time length, and a log detection result of each sub-log data stream is obtained.

[0067] In one embodiment, the real-time consumption of the log data in the message queue by using flink to obtain the log data stream of each piece of log data of each data interface comprises: detecting the log data of each data interface to determine whether real log data is missing in any one data interface within the detection time length; in the case that the real log data is missing, generating a preset number of simulated log data to supplement the log data of the data interface in which the real log data is missing, so as to obtain the log data stream of each piece of log data of each data interface.

[0068] In one embodiment, the determining the monitoring analysis data of each data interface according to the monitoring rule data stream contained in the fusion data stream comprises: transmitting the fusion data stream to a detection window to determine a data transmission success rate of each sub-log data stream, and storing the data transmission success rate to a ring queue; determining a monitoring time period of each data interface according to the monitoring rule data stream contained in the fusion data stream; determining, for each data interface, a first number of detection time lengths required to reach the monitoring time period of the data interface; selecting, for each data interface, the data transmission success rates of the first number of sub-log data streams corresponding to the data interface from the ring queue in chronological order; and determining, for each data interface, the monitoring analysis data of the data interface according to the data transmission success rates of the first number of sub-log data streams.

[0069] In one embodiment, the interface monitoring rule comprises a success rate range in each monitoring time period, and the determining the monitoring analysis data of each data interface according to the data transmission success rates of the first number of sub-log data streams comprises: determining an average value of the data transmission success rates of the first number of sub-log data streams; in a case where the average value is within the success rate range, determining that the log data stream corresponding to the data interface meets the interface monitoring rule of the data interface; and in a case where the average value is not within the success rate range, determining that the log data stream corresponding to the data interface does not meet the interface monitoring rule of the data interface.

[0070] In one embodiment, the data processing method further comprises: in a case where it is determined that the log data stream corresponding to the data interface meets the interface monitoring rule of the data interface, obtaining feedback information for the data interface; in a case where the feedback information is transmission abnormal, determining that the interface monitoring rule is to be modified; and in a case where the feedback information is transmission normal, determining that the data transmission of the data interface is normal.

[0071] In one embodiment, the real-time consumption of the interface monitoring rule stored in the redis to obtain the monitoring rule data stream of each data interface comprises: in a case where the interface monitoring rule is to be modified, obtaining the updated interface monitoring rule, and performing real-time broadcasting of the updated interface monitoring rule in the flink cluster to cover the interface monitoring rule before the update.

[0072] Those skilled in the art will understand that the embodiments of the present application can be provided as a method, a system, or a computer program product. Therefore, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0073] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks. Figure 1 one or more flowcharts and / or blocks Figure 1 means for functionally implementing the steps listed in the flowchart block or blocks.

[0074] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instructions which implement the function specified in the flowchart block or blocks. Figure 1 one or more flowcharts and / or blocks Figure 1 means for functionally implementing the steps listed in the flowchart block or blocks.

[0075] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks. Figure 1 one or more flowcharts and / or blocks Figure 1 means for functionally implementing the steps listed in the flowchart block or blocks.

[0076] In a typical configuration, a computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0077] The memory can include non-persistent memory and / or volatile memory, such as a random access memory (RAM) including a cache area for the temporary storage of data. The memory can also include non-volatile memory, such as read only memory (ROM), electrically programmable read only memory (EPROM), or electrically erasable programmable read only memory (EEPROM), for the storage of software that is read during runtime. The memory is an example of computer readable media.

[0078] Computer-readable media includes permanent and non-permanent, movable and non-movable media that can implement information storage by any method or technology. The information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassette, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information accessible to a computing device. According to the definition herein, computer-readable media does not include transitory media such as modulated data signals and carriers.

[0079] It should also be noted that the terms "comprising", "containing", or any other variant thereof are intended to cover a non-exclusive inclusion, such that a process, method, article or apparatus that comprises a list of elements does not only include those elements, but can also include other elements not expressly listed or inherent to such process, method, article or apparatus. Without more limitations, the element defined by the statement "comprising a" does not exclude the presence of additional identical elements in the process, method, article or apparatus that includes the element.

[0080] The above is only an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application can have various modifications and changes. Any modification, equivalent replacement, improvement, etc. within the spirit and principle of the present application shall be included in the scope of claims of the present application.

Claims

1. A data processing method, characterized in that, The data processing method, applied to a data service platform, includes: Retrieve the interface monitoring rules configured for each data interface; Determine the greatest common divisor of the monitoring time periods of all data interfaces based on the monitoring time period of each data interface; The greatest common divisor is determined as the detection time length of the detection window for the log data stream in the fused data stream; wherein, when analyzing the log data stream through the detection window, the log data stream is divided into multiple sub-log data streams according to the detection time length, and the log detection result of each sub-log data stream is obtained; The interface monitoring rules are saved to Redis, and the interface monitoring rules include the monitoring time period; Log data from each data interface is acquired in real time and stored in a message queue. The log data in the message queue is consumed in real time using Flink to obtain a log data stream for each log data from each data interface; Consume the interface monitoring rules stored in Redis in real time to obtain the monitoring rule data stream for each data interface; The log data stream and monitoring rule data stream of each data interface are merged to obtain the merged data stream of each data interface; The fused data stream is analyzed based on the monitoring rule data stream contained therein to obtain monitoring analysis data for each data interface; The real-time consumption of log data in the message queue using Flink to obtain a log data stream for each log data item from each data interface includes: The log data of each data interface is inspected to determine whether any data interface has experienced a real log data loss within the specified inspection time period. In the event of missing real log data, a preset number of simulated log data are generated to supplement the log data of the data interface that is missing real log data, thereby obtaining the log data stream of each log data of each data interface. The step of analyzing the fused data stream based on the monitoring rule data stream contained in the fused data stream to obtain monitoring analysis data for each data interface includes: The fused data stream is transmitted to the detection window to determine the data transmission success rate of each sub-log data stream, and the data transmission success rate is stored in a circular queue. The monitoring time period for each data interface is determined based on the monitoring rule data stream contained in the fused data stream; For each data interface, determine a first number of detection time lengths required to reach the monitoring time period of the data interface; For each data interface, the data transmission success rate of the first number of sub-log data streams corresponding to the data interface is selected from the circular queue in chronological order. For each data interface, the monitoring and analysis data of the data interface is determined based on the data transmission success rate of the first number of sub-log data streams.

2. The data processing method according to claim 1, characterized in that, The interface monitoring rules include a success rate range for each monitoring time period. For each data interface, determining the monitoring and analysis data based on the data transmission success rate of the first number of sub-log data streams includes: Determine the average data transmission success rate of the first number of sub-log data streams; If the average value is within the success rate range, it is determined that the log data stream corresponding to the data interface conforms to the interface monitoring rules of the data interface. If the average value is not within the success rate range, it is determined that the log data stream corresponding to the data interface does not conform to the interface monitoring rules of the data interface.

3. The data processing method according to claim 2, characterized in that, The data processing method further includes: If it is determined that the log data stream corresponding to the data interface conforms to the interface monitoring rules of the data interface, feedback information for the data interface is obtained; If the feedback information indicates a transmission anomaly, it is determined that the interface monitoring rules need to be modified. If the feedback information indicates that the transmission is normal, it is determined that the data transmission of the data interface is normal.

4. The data processing method according to claim 3, characterized in that, The real-time consumption of the interface monitoring rules stored in Redis to obtain the monitoring rule data stream for each data interface includes: Once it is determined that the interface monitoring rule needs to be modified, and the updated interface monitoring rule is obtained, the updated interface monitoring rule is broadcast in real time in the Flink cluster to override the interface monitoring rule before the update.

5. A processor, characterized in that, It is configured to perform the data processing method according to any one of claims 1 to 4.

6. A data processing apparatus, characterized in that, Includes the processor as described in claim 5.

7. A machine-readable storage medium storing instructions thereon, characterized in that, When executed by a processor, this instruction causes the processor to be configured to perform the data processing method according to any one of claims 1 to 4.

Images