Data processing method, apparatus and system

Abstract

The application discloses a data processing method, device and system, and aims to solve the problem of poor data processing reliability in related monitoring data processing technology. The method comprises the following steps: acquiring to-be-processed data, and recording a first timestamp of acquiring the to-be-processed data; sending the to-be-processed data and the first timestamp corresponding to the to-be-processed data to a check queue as to-be-confirmed data, and sending the to-be-processed data to a data processing service cluster to enable the data processing service cluster to process the to-be-processed data; determining whether there is target to-be-confirmed data in the check queue based on a first current time, a preset timeout threshold and the first timestamp; to-be-confirmed data corresponding to to-be-processed data that has been processed is removed from the check queue; if there is target to-be-confirmed data, the to-be-processed data corresponding to the target to-be-confirmed data is reprocessed, and the target to-be-confirmed data is removed from the check queue.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of data processing, and particularly relates to a data processing method, device and system. BACKGROUND

[0002] The related fifth generation mobile communication technology (5th Generation Mobile Communication Technology, 5G) messages, instant messaging (Instant Messaging, IM) and other information and communications technology (information and communications technology, ICT) architecture services often use monitoring systems such as Zabbix and Prometheus as system and network monitoring tools.

[0003] Taking zabbix as an example, the monitoring data collection, preprocessing, analysis and alarm triggering are systematic, the server+proxy mode is used to support good scalability, the monitoring data collection of large-scale service groups can be applied, and the monitoring data collection is widely applied to various application platforms. The monitoring data is collected by the agent software agent deployed by the monitored object, is summarized to the zabbix server for analysis and processing, is stored in the database, and is monitored and visualized by the zabbix network web component. In a large-scale data scenario, the zabbix server processes and caches data through an open-source high-performance key-value storage database redis or a message queue middleware to avoid excessive instantaneous connection numbers of the data processing node and the database, and to cause blockage of upstream and downstream service processes.

[0004] However, after the data is taken out from the queue and sent to the distributed data processing node, until the data is written into the database, there is a lack of persistent storage and copy protection, in the case that the processing thread abnormally ends, the processing process is false death and restarts, the processing node abnormally offlines and the like, the unprocessed data is lost and cannot be recovered.

[0005] That is, the related monitoring data processing technology has the problem of poor reliability of data processing. SUMMARY

[0006] The embodiments of the application provide a data processing method, device and system, which can solve the problem of poor reliability of data processing of the related monitoring data processing technology.

[0007] In a first aspect, the embodiments of the present application provide a data processing method, which comprises: obtaining to-be-processed data, and recording a first timestamp of obtaining the to-be-processed data; sending the to-be-processed data and the first timestamp corresponding to the to-be-processed data as to-be-confirmed data to a check queue, and sending the to-be-processed data to a data processing service cluster, so that the data processing service cluster processes the to-be-processed data; determining, by a check controller, whether there is target to-be-confirmed data in the check queue based on a first current time, a preset timeout threshold and the first timestamp; wherein the target to-be-confirmed data is the to-be-confirmed data corresponding to the to-be-processed data that has timed out; the to-be-confirmed data corresponding to the to-be-processed data that has been processed is removed from the check queue; if the target to-be-confirmed data exists, the to-be-processed data corresponding to the target to-be-confirmed data is reprocessed, and the target to-be-confirmed data is removed from the check queue.

[0008] In a second aspect, the embodiments of the present application provide a data processing system, which comprises: a data collection queue cluster, configured to obtain to-be-processed data, and record a first timestamp of obtaining the to-be-processed data; send the to-be-processed data and the first timestamp corresponding to the to-be-processed data as to-be-confirmed data to a check queue, and send the to-be-processed data to a data processing service cluster; the data processing service cluster, configured to process the to-be-processed data, and reprocess the to-be-processed data corresponding to target to-be-confirmed data; wherein the target to-be-confirmed data is the to-be-confirmed data corresponding to the to-be-processed data that has timed out; the check queue, configured to store the to-be-confirmed data; the check controller, configured to remove the to-be-confirmed data corresponding to the to-be-processed data that has been processed and the target to-be-confirmed data from the check queue, and determine whether the target to-be-confirmed data exists in the check queue based on a first current time, a preset timeout threshold and the first timestamp.

[0009] In a third aspect, an embodiment of the present application provides a data processing apparatus, the apparatus comprising: an obtaining module configured to obtain to-be-processed data and record a first timestamp of obtaining the to-be-processed data; a sending module configured to send the to-be-processed data and the first timestamp corresponding to the to-be-processed data as to-be-confirmed data to a check queue, and send the to-be-processed data to a data processing service cluster to enable the data processing service cluster to process the to-be-processed data; a determining module configured to determine whether there is target to-be-confirmed data in the check queue based on a first current time, a preset timeout threshold, and the first timestamp corresponding to to-be-confirmed data in the check queue, wherein the target to-be-confirmed data is to-be-confirmed data corresponding to the to-be-processed data that has timed out, and to-be-confirmed data corresponding to the to-be-processed data that has been processed is removed from the check queue; and a reprocessing module configured to reprocess the to-be-processed data corresponding to the target to-be-confirmed data and remove the target to-be-confirmed data from the check queue if the target to-be-confirmed data exists.

[0010] In a fourth aspect, an embodiment of the present application provides an electronic device, comprising: a processor; and a memory arranged to store computer executable instructions, the executable instructions being configured to be executed by the processor, and the executable instructions comprising instructions for performing the data processing method according to the first aspect.

[0011] In a fifth aspect, an embodiment of the present application provides a storage medium for storing computer executable instructions, the computer executable instructions enabling a computer to perform the data processing method according to the first aspect.

[0012] In a sixth aspect, an embodiment of the present application provides a chip, the chip comprising a processor and a communication interface, the communication interface being coupled to the processor, and the processor being configured to run a program or instructions to implement the data processing method according to the first aspect.

[0013] In a seventh aspect, an embodiment of the present application provides a computer program product comprising a computer program, the computer program being executed by a processor to implement the data processing method according to the first aspect.

[0014] In the embodiment of the present application, the to-be-processed data is first acquired, and a first timestamp of acquiring the to-be-processed data is recorded; the to-be-processed data and the first timestamp corresponding to the to-be-processed data are sent to a check queue as to-be-confirmed data, and the to-be-processed data is sent to a data processing service cluster to enable the data processing service cluster to process the to-be-processed data; then, based on a first current time, a preset timeout threshold and the first timestamp, the check controller determines whether there is target to-be-confirmed data in the check queue; the target to-be-confirmed data is to-be-confirmed data corresponding to to-be-processed data that has timed out; to-be-confirmed data corresponding to to-be-processed data that has been processed is removed from the check queue; then, if there is target to-be-confirmed data, the to-be-processed data corresponding to the target to-be-confirmed data is reprocessed, and the target to-be-confirmed data is removed from the check queue. Compared with related monitoring data processing technologies that lack persistent storage and copy protection, the present application adds a check queue and a check controller to implement a storage mechanism for to-be-confirmed data corresponding to to-be-processed data and a check mechanism for data processing, so that when an abnormal event occurs to cause to-be-processed data that has not been processed to appear, the to-be-processed data that has not been processed is found and reprocessed, thereby avoiding loss of to-be-processed data and the inability to recover and process the to-be-processed data, and thus improving the reliability of processing to-be-processed data and solving the problem of poor reliability of data processing in related monitoring data processing technologies. BRIEF DESCRIPTION OF DRAWINGS

[0015] Figure 1 is a flow diagram of a data processing method provided by an embodiment of the present application;

[0016] Figure 2 is a structural diagram of a workgroup provided by an embodiment of the present application;

[0017] Figure 3 is a diagram of classification of processing conditions of to-be-processed data provided by an embodiment of the present application;

[0018] Figure 4 is a flow diagram of another data processing method provided by an embodiment of the present application;

[0019] Figure 5 is a schematic block diagram of a data processing system provided by an embodiment of the present application;

[0020] Figure 6 is a schematic block diagram of another data processing system provided by an embodiment of the present application;

[0021] Figure 7 is a structural diagram of a data processing apparatus provided by an embodiment of the present application;

[0022] Figure 8Fig. 1 is a structural schematic diagram of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0023] The technical solutions in the embodiments of the present application will be clearly and completely described in connection with the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of protection of the present application.

[0024] The terms "first", "second", and the like in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described herein, and the objects distinguished by "first", "second", etc. are generally of a kind and do not limit the number of objects, for example, the first object can be one or more. In addition, "and / or" in the specification and claims indicates at least one of the connected objects, and the character " / ", generally indicates that the objects before and after are in an "or" relationship.

[0025] The data processing method, device and system provided by the embodiments of the present application will be described in detail below in connection with the drawings, through specific embodiments and their application scenarios.

[0026] Figure 1 An embodiment of the present application provides a data processing method, which can be executed by an electronic device, which can include a server and / or a terminal device, for example, a vehicle-mounted terminal or a mobile phone terminal. In other words, the method can be executed by software or hardware installed in the electronic device, and the method includes the following steps:

[0027] S102: Obtain the to-be-processed data, and record a first timestamp of obtaining the to-be-processed data.

[0028] In actual application, the business cluster can be monitored to collect monitoring data; then, the monitoring data is sent to a data collection queue cluster, and the data collection queue cluster generates to-be-processed data corresponding to the monitoring data. Then, in the case that a data processing service cluster sends a data processing consumption request, the to-be-processed data is taken out, and the time of taking out the to-be-processed data is recorded as the first timestamp. Specifically, an agent can be installed in the business cluster host to be monitored, so as to collect the monitoring data M.

[0029] S104: send the to-be-processed data and the first timestamp corresponding to the to-be-processed data to the check queue as to-be-confirmed data, and send the to-be-processed data to the data processing service cluster to enable the data processing service cluster to process the to-be-processed data.

[0030] The to-be-confirmed data can be sent to the check queue, and the to-be-processed data can be sent to the data processing service cluster.

[0031] S106: determine, by the check controller, whether there is target to-be-confirmed data in the check queue based on the first current time, the preset timeout threshold, and the first timestamp.

[0032] The target to-be-confirmed data is to-be-confirmed data corresponding to to-be-processed data that has timed out, and to-be-confirmed data corresponding to to-be-processed data that has been processed is removed from the check queue.

[0033] In actual application, the check controller and the check queue can be in a data check queue cluster.

[0034] Specifically, when the difference between the first current time (hereinafter referred to as T) and the first timestamp (hereinafter referred to as T1) corresponding to the to-be-confirmed data is greater than or equal to the preset timeout threshold (hereinafter referred to as D), it is determined that the to-be-confirmed data is to-be-confirmed data corresponding to to-be-processed data that has timed out. The first current time is the time at which the check controller determines whether there is target to-be-confirmed data.

[0035] In actual application, to-be-confirmed data corresponding to to-be-processed data that has been processed is removed from the check queue, and target to-be-confirmed data (to-be-confirmed data corresponding to to-be-processed data that has timed out) is also removed. The to-be-confirmed data at the head of the check queue is always to-be-confirmed data that has not been processed and that was processed earliest, so when determining whether to-be-processed data corresponding to to-be-confirmed data in the check queue has timed out, in the case that the to-be-confirmed data at the head of the check queue has not timed out, it is generally sufficient to determine whether to-be-processed data corresponding to the to-be-confirmed data at the head of the check queue has timed out, without determining whether to-be-processed data corresponding to other to-be-confirmed data in the check queue has timed out.

[0036] S108: if there is target to-be-confirmed data, reprocess to-be-processed data corresponding to the target to-be-confirmed data, and remove the target to-be-confirmed data from the check queue.

[0037] Specifically, the to-be-processed data (the to-be-processed data corresponding to the target to-be-confirmed data) whose processing is overdue can be sent back to the data collection queue cluster, so that the to-be-processed data whose processing is overdue is reprocessed, such as re-executing S102 to S108 on the monitoring data whose processing is overdue. The target to-be-confirmed data can be removed from the inspection queue by the inspection controller.

[0038] In addition, the to-be-confirmed data corresponding to the to-be-processed data whose processing is overdue in the inspection queue can be removed after the to-be-processed data whose processing is overdue is sent back to the data collection queue cluster.

[0039] The data processing method provided by the embodiment of the application comprises the following steps: obtaining to-be-processed data, and recording a first time stamp of the to-be-processed data; sending the to-be-processed data and the first time stamp corresponding to the to-be-processed data to an inspection queue as to-be-confirmed data, and sending the to-be-processed data to a data processing service cluster to enable the data processing service cluster to process the to-be-processed data; determining, by an inspection controller, whether there is target to-be-confirmed data in the inspection queue based on a first current time, a preset timeout threshold and the first time stamp, wherein the target to-be-confirmed data is to-be-confirmed data corresponding to to-be-processed data whose processing is overdue, and to-be-confirmed data corresponding to to-be-processed data that has been processed completely is removed from the inspection queue; and if the target to-be-confirmed data exists, reprocessing the to-be-processed data corresponding to the target to-be-confirmed data, and removing the target to-be-confirmed data from the inspection queue. Compared with related monitoring data processing technologies that lack persistent storage and copy protection, the present application adds an inspection queue and an inspection controller to realize a storage mechanism for to-be-confirmed data corresponding to to-be-processed data and an inspection mechanism for data processing, so that to-be-processed data that has not been processed completely can be found and reprocessed when an abnormal event occurs, thereby avoiding loss of to-be-processed data and improving the reliability of processing to-be-processed data.

[0040] In an implementation manner, the to-be-processed data (i.e., S102) can be obtained, which can be specifically implemented as steps A1 to A3:

[0041] Step A1: obtaining monitoring data by a collection controller in a data collection queue cluster, and assigning the monitoring data to a pre-set work group.

[0042] The work group comprises data processing nodes in a data processing service cluster.

[0043] Due to the distributed nature of the data processing service cluster, the performance, health, and stability of different nodes are different. Each unit node participating in distributed data processing in the data processing service cluster can be managed as a data processing node. A unique (hereinafter) collection queue node, (hereinafter) data inspection node, and (hereinafter) running statistics node can be set for each data processing node, thereby forming multiple work groups (pre-set work groups) and processing monitoring data in units of work groups.

[0044] Specifically, the data processing node includes a processing controller and a thread pool, and the thread pool includes multiple threads.

[0045] Step A2, through the collection controller, based on the monitoring data and the work group to which the monitoring data is assigned, generates to-be-processed data, and stores the to-be-processed data in the data collection queue cluster.

[0046] Specifically, through the collection controller, based on the monitoring data M and the work group to which the monitoring data is assigned, a globally unique data identifier M ID is generated. ID The data identifier and the monitoring data are used to identify the monitoring data and the work group to which the monitoring data is assigned. Then, the to-be-processed data can be stored in the data collection queue cluster in the form of a two-tuple (M ID , M). Correspondingly, the to-be-confirmed data can be (M ID , M, T1).

[0047] Exemplarily, M ID may be a 32-byte globally unique data identifier, of which the first 8 bytes are used to identify the work group. In the above S106, the to-be-confirmed data corresponding to the to-be-processed data that has been processed is removed from the inspection queue. Specifically, the to-be-confirmed data can be removed from the inspection queue by matching the first 8 bytes of M ID in the to-be-processed data (M ID , M) to the inspection queue corresponding to the work group, and then matching the to-be-confirmed data with the same data identifier M ID in the inspection queue corresponding to the work group.

[0048] It should be noted that the above M ID is 32 bytes, of which the first 8 bytes are used to identify the work group, which is only an example and does not limit the data identifier.

[0049] Correspondingly, the reprocessing of the target to-be-confirmed data in S108 can be specifically performed as follows: the monitoring data M corresponding to the to-be-processed data that exceeds the processing time limit is sent to the collection controller again by the inspection controller, so that the collection controller reassigns a work group to the monitoring data M; or the to-be-processed data that exceeds the processing time limit is directly sent to the original work group of the data collection queue cluster; and then the data processing node in the work group processes the to-be-processed data.

[0050] In step A3, the data collection queue cluster acquires the to-be-processed data stored in the data collection queue cluster.

[0051] Correspondingly, the sending of the to-be-processed data to the data processing service cluster for processing (S104) can be specifically performed as step A4.

[0052] In step A4, the to-be-processed data is sent to the data processing node corresponding to the work group, so that the data processing node corresponding to the work group processes the to-be-processed data.

[0053] In this embodiment, the work groups are pre-set, and the monitoring data is assigned to appropriate work groups, so that the reprocessing of the monitoring data is more reasonable.

[0054] In an implementation manner, the target module can be updated by performing the following step B1.

[0055] In step B1, the running parameter information of each work group pre-stored in the running statistics database is pushed to the target module by the running parameter controller, so that the target module is updated based on the running parameter information of each work group.

[0056] The running parameter information is related to the historical processing of the to-be-processed data by the work group.

[0057] The target module includes one or more of the following: the collection controller and the inspection controller.

[0058] In this embodiment, the target module of each work group is updated based on the running parameter information (related to the historical processing of the to-be-processed data by the work group) of each work group by the inspection controller, feedback control is realized, and the reliability and controllability of the monitoring data collection and processing process are improved.

[0059] In an implementation manner, the work group further includes a data inspection node, and the data inspection node includes the inspection queue. The inspection controller determines whether the target to-be-confirmed data exists in the inspection queue based on the first current time, the preset timeout threshold, and the first timestamp (S106), which can be specifically performed as step C1.

[0060] Step C1, by the inspection controller, based on the first current time, the preset timeout threshold corresponding to the work group and the first timestamp, determine whether there is target to be confirmed data in the inspection queue.

[0061] Optionally, the preset timeout threshold corresponding to different work groups can be the same or different.

[0062] If the target module includes an inspection controller, the target module updates the running parameter information of each work group accordingly (i.e., step B1), which can be specifically performed as step b1:

[0063] Step b1, by the inspection controller, based on the running parameter information of each work group and the preset timeout rate, update the preset timeout threshold of each work group.

[0064] Specifically, the running parameter information can include the probability of timeout processing completion and the average timeout time of timeout completion within a preset period. Based on the probability of timeout processing completion and the average timeout time of timeout completion within a preset period and the preset timeout rate , determine the new preset timeout threshold :

[0065]

[0066] Among them, PO is the probability of timeout processing completion within a preset period, and EO is the average timeout time of timeout completion.

[0067] In this embodiment, by the inspection controller, based on the running parameter information of each work group (related to the historical processing situation of the work group to the to-be-processed data) and the preset timeout rate, the preset timeout threshold of each work group is updated, feedback control is realized, and on the basis of setting the inspection mechanism for the data processing situation, the preset timeout threshold for the inspection mechanism can be dynamically adjusted according to the running state of the work group, and dynamic inspection of data processing is realized. And setting the corresponding preset timeout threshold for different work groups can make more reasonable judgment on whether the monitored data is processed overtime, so that the reprocessing of the monitored data is more reasonable.

[0068] In one implementation mode, the pre-set work group is allocated to the monitored data (i.e., step A1), which can be specifically performed as step a1:

[0069] Step a1, by the collection controller, based on the preset weight of each work group, the preset total weight and the preset running weight of each work group, using a load balancing algorithm, determine the work group allocated to the monitored data.

[0070] Among them, the load balancing algorithm can be a smooth weighted round robin method.

[0071] Specifically, in the initial stage: ;

[0072] Among them, WC i W is the preset operating weight for each working group. i S represents the preset weights for each working group. Wi This is the preset total weight.

[0073] Whenever the data acquisition controller receives new monitoring data, it assigns workgroups to the monitoring data through steps i to iii, and updates the preset operating weights of each workgroup:

[0074] Step i, upon receiving new monitoring data M, set the preset operating weights of each workgroup to WC. i新 =WC i +W i .

[0075] Step ii, iterate through the preset running weights WC of each workgroup. i新 Find the WC i新 The group g with the largest value, the WC of group g i新 That is, WC g .

[0076] Step iii: Assign the new monitoring data M to the workgroup g, and instruct WC to... g新 =WC g -S Wi .

[0077] If the target module includes a data acquisition controller, the target module updates accordingly based on the operating parameter information of each working group (i.e., step B1), and can specifically execute step b2:

[0078] Step b2: By collecting data from the controller, update the preset weights and preset total weights of each working group based on the operating parameter information of each working group.

[0079] Specifically, the operating parameters for each working group should include at least the following: the number of data processing messages with a result category of the first category within multiple preset periods, the number of data processing messages with a result category of the second category, the probability of completion within a preset timeout threshold, and the average processing completion time. The preset periods can be 1 minute, 3 minutes, 15 minutes, etc.

[0080] Based on the operating parameter information of each working group, the preset weights and preset total weights of each working group are updated as follows:

[0081]

[0082] in, CO1 is the number of data processing information whose processing result category is the first category (indicating that the to-be-processed data is processed within the preset timeout threshold) in a preset period of 3 minutes; CO3 is the number of data processing information whose processing result category is the second category (indicating that the to-be-processed data is processed beyond the timeout) in a preset period of 3 minutes; is the probability of being processed within the preset timeout threshold in a preset period of 15 minutes; is the probability of being processed within the preset timeout threshold in a preset period of 3 minutes; is the average processing completion time in a preset period of 3 minutes.

[0083] In this embodiment, the preset weights and the preset total weight of each workgroup are updated based on the running parameter information (related to the historical processing of the to-be-processed data by the workgroup) of each workgroup by the collection controller, feedback control is achieved, and on the basis of the workgroup determined by the load balancing algorithm for monitoring data distribution, the to-be-processed data can be dynamically distributed to different workgroups according to the running state of the workgroup, and dynamic load balancing is achieved. Compared with the relatively fixed parameter load balancing method of the related scheme, it is more suitable for situations such as uneven performance of data processing nodes and load fluctuations, and has stronger flexibility.

[0084] In an implementation manner, the data collection queue cluster further includes a collection queue node; the workgroup further includes a collection queue node; the collection queue node includes a flow limiter and a collection queue; and the workgroup can be specifically as shown in Figure 2 The to-be-processed data is stored in the data collection queue cluster (i.e., step A2), which can be specifically performed as follows.

[0085] Step D1, storing the to-be-processed data in the collection queue corresponding to the workgroup.

[0086] The to-be-processed data is obtained, and a first timestamp of obtaining the to-be-processed data is recorded (i.e., S102), which can be specifically performed as follows.

[0087] Step D2, through the flow limiter of the workgroup, the to-be-processed data is taken out from the collection queue corresponding to the workgroup according to the time interval determined by the flow limiter, and a first timestamp of taking out the to-be-processed data from the collection queue is recorded.

[0088] The collection queue is a buffer zone before data processing, accumulates to-be-processed data, and the data processing node corresponding to the workgroup consumes data from the collection queue for processing.

[0089] In actual application, the data processing node can actively initiate a data processing consumption request to the collection queue node, the flow limiter receives the data processing consumption request, and takes out the to-be-processed data from the collection queue according to the time interval determined by the flow limiter.

[0090] The data collection queue cluster sends the to-be-processed data and the first timestamp corresponding to the to-be-processed data as to-be-confirmed data to the check queue, and sends the to-be-processed data to the data processing service cluster, so that the data processing service cluster processes the to-be-processed data (that is, step S104), which can be specifically implemented as the following step D3:

[0091] In step D3, the flow limiter is added to the collection queue in the work group to implement the frequency control and flow control mechanism. Compared with the mode of a small amount of host resource exclusive and a large amount of host resource multiplexing used in related distributed monitoring systems, because the data processing node rate is usually lower than the data queue, in the case of sufficient load during the business peak period, the exclusive host will continue to run at the maximum performance, which affects the service life and reliability of the equipment. The multiplexed host will have a greater impact on the performance of other multiplexed services as the load rises. The embodiment realizes the active control of the data processing node load at the queue level, and can reserve computing resources and avoid continuous maximum load operation in the equipment multiplexing and equipment recycling scenarios.

[0092] In an implementation manner, the to-be-processed data is taken out from the collection queue according to the time interval determined by the flow limiter (that is, step D2), which can be specifically implemented as the following step d1:

[0093] In step d1, the time interval is determined based on the preset flow control frequency by the flow limiter, and the to-be-processed data is taken out from the collection queue according to the time interval.

[0094] Specifically, in the initialization stage, the preset flow control frequency of the collection queue of each work group can be artificially set The flow limiter of each work group can control the time interval of consuming data of adjacent two collection queues with different methods according to different intervals of the preset flow control frequency with 1 second as a control period.

[0095] When the query rate per second (QPS, Queries-per-second) is less than or equal to the preset flow control frequency, the following steps I to III are cyclically executed: In step I, the current time T is recorded, and a to-be-processed data Mj is read from the collection queue, and a data processing consumption request is waited for;

[0096] In step II, after receiving the data processing consumption request, the data processing consumption request is responded to, and the to-be-processed data Mj is removed from the collection queue;

[0097] In step III, the first timestamp T1 of taking out the to-be-processed data from the collection queue is recorded, the time interval

[0098] is calculated, and the d duration is waited for. ​

[0099] When , the time interval of taking out the data to be processed is dynamically controlled, the current control period S=0 is initialized, the number of data processing consumption requests CR in the current control period is initialized, and the following steps ① to ④ are executed in a loop:

[0100] Step ①, record the current time T, and check whether the current control period is over:

[0101] When , set CR=0, , otherwise, when , set CR=0, ;

[0102] Step ②, read a piece of data to be processed Mj from the collection queue, and wait for a data processing consumption request;

[0103] Step ③, after receiving a data processing consumption request, respond to the data processing consumption request, increment CR by one, and remove Mj from the collection queue in parallel;

[0104] Step ④, record the first timestamp T1 of taking out the data to be processed from the collection queue, calculate the time interval d using the following formula, and wait for d duration:

[0105]

[0106] When , the time interval of taking out the data to be processed in each control period is dynamically controlled, the current control period S=0 is initialized, and the following steps 1) to 3) are executed in a loop:

[0107] Step 1), record the current timestamp T, and update the current control period. When , set ;

[0108] Step 2), read pieces of data to be processed from the collection queue at a time, continuously respond to data processing consumption requests, and remove the data to be processed from the collection queue after all responses are completed;

[0109] Step 3), record the first timestamp T1 of taking out the data to be processed from the collection queue, calculate the time interval d as follows, and wait for d duration:

[0110]

[0111] The above target module also includes a flow limiter, which updates based on the running parameter information of each working group (i.e., step B1), which can specifically execute step b3:

[0112] Step b3, updating the preset throttling frequency of each workgroup based on the operation parameter information of each workgroup by the throttler.

[0113] Specifically, the operation parameter information includes a probability of loss of the to-be-processed data within a preset period.

[0114] The updating of the preset throttling frequency of each workgroup based on the operation parameter information of each workgroup can be performed according to the following formula:

[0115]

[0116] wherein, is the probability of loss of the to-be-processed data within the preset period.

[0117] In this embodiment, the preset throttling frequency of each workgroup is updated based on the operation parameter information (related to the historical processing of the to-be-processed data by the workgroup) of each workgroup by the throttler, feedback control is realized, and on the basis of adding the frequency control throttling mechanism to the collection queue, the preset throttling frequency can be dynamically adjusted according to the operation state of the workgroup, dynamic control of the consumption rate is realized. Moreover, different control methods are set for different preset throttling frequency intervals.

[0118] In an implementation manner, the workgroup further includes an operation statistics node, and the operation statistics node includes a historical queue; the historical queue is configured to store data record information corresponding to the to-be-confirmed data removed from the inspection queue corresponding to the workgroup by the inspection controller; the data record information includes to-be-processed data, a first timestamp corresponding to the to-be-processed data, a second timestamp, and a preset timeout threshold; the second timestamp represents a time when the to-be-processed data is processed; and the operation parameter information of each workgroup is determined through the following steps E1 to E2:

[0119] Step E1, determining, by the statistical controller, a processing result category of the data record information in the historical queue based on the second current time, the data record information stored in the historical queue corresponding to the workgroup, and a preset timeout tolerance value.

[0120] The processing result category includes a first category, a second category, and a third category; the first category represents that the to-be-processed data is processed within the preset timeout threshold; the second category represents that the to-be-processed data is processed overtime; and the third category represents that the to-be-processed data is lost.

[0121] In actual application, the second current time and the first current time can be the same or different, and the values of the two are determined by the time of determining the processing result category and the time of determining whether there is target to-be-confirmed data, respectively.

[0122] Specifically, the data record information corresponding to the to-be-confirmed data removed from the inspection queue by the inspected controller stored in the history queue includes the data record information corresponding to the to-be-processed data processed and the data record information corresponding to the to-be-processed data processed overtime.

[0123] Considering that there are abnormal events such as node hardware or system failure, exhaustion of computing or storage resources, too high network latency, termination of service threads, etc., which cause the data processing task to be interrupted irrecoverably, and thus unable to be completed within the preset timeout tolerance value Dmax, i.e., data loss; there are also temporary reasons such as CPU busy, performance bottleneck, resource shortage, network flash-off, etc., which cause the data processing task to be abnormally delayed and unable to be processed within the preset timeout threshold D, but completed within the preset timeout tolerance value Dmax, i.e., timeout processing completion. Therefore, as shown in FIG. 1, the processing condition (processing result category) of the to-be-processed data is at least divided into three categories by the statistics controller. Figure 3

[0124] Specifically, the data record information can be (M ID , M, T1, T2, D). First, the processing completion delay TF = T2-T1 of each data record information is determined according to the difference between the second timestamp (T2) and the first timestamp (T1) in the data record information, and whether the to-be-processed data corresponding to the data record information is processed or not is determined according to whether the processing completion delay is positive or not.

[0125] If TF<0 (not processed), the second current time T is recorded, and whether the to-be-processed data is lost is determined according to whether T-T1 is greater than the preset timeout tolerance value Dmax or not; if T-T1 is greater than the preset timeout tolerance value, the processing result category R of the data record information is determined as the third category; if T-T1 is less than the preset timeout tolerance value, the data record information is skipped and no processing is performed thereon.

[0126] If TF>0 (processed), whether the to-be-processed data is processed overtime is determined according to whether the processing completion delay TF is greater than or equal to the preset timeout threshold D or not; if TF is greater than or equal to the preset timeout threshold, the processing result category R of the data record information is determined as the second category; if TF is less than the preset timeout threshold, the processing result category R of the data record information is determined as the first category.

[0127] Step E2, the data record information and the processing result category are stored in the running statistics database as data processing information by the statistics controller, and the running parameter information of each working group is determined based on the data processing information in the running statistics database.

[0128] Specifically, the data processing information can be (M ID ​, M, T1, T2, D, R), and stores the data processing information into the running statistics database. In addition, the data record information in the historical queue which has determined the processing result category is deleted. Then, the statistical controller can determine the running parameter information of each work group based on the data processing information in the running statistics database according to the preset trigger interval and in the unit of work group.

[0129] In the embodiment, the data record information is obtained by setting the historical queue to connect the inspection queue, and the running parameter information of each work group is further determined based on the set statistical controller. Further, by setting the historical queue, the inspection queue and the collection queue, a three-level queue mechanism is formed to realize the real-time monitoring and recording of the data processing running state of each work group.

[0130] In an implementation manner, before determining the processing result category of the data record information in the historical queue based on the second current time, the data record information stored in the historical queue corresponding to the work group and the preset timeout tolerance value by the statistical controller (i.e., S104), the data record information sent into the historical queue can be determined by performing the following steps F1 to F2:

[0131] In step F1, if the message representing that the to-be-processed data is processed is received by the inspection controller, the third current time is taken as the second time stamp of the processed to-be-processed data, and the processed to-be-processed data, the second time stamp, the first time stamp and the preset timeout threshold value are taken as the data record information and sent into the historical queue of the work group.

[0132] In the embodiment, the third current time is the time when the message representing that the to-be-processed data is processed is received.

[0133] Specifically, when the message indicating that the to-be-processed data is processed is received, there are two cases: in the first case, the to-be-processed data is processed within the preset timeout threshold, and accordingly, the check controller can match the to-be-confirmed data corresponding to the to-be-processed data in the check queue; in the second case, the to-be-processed data is processed overtime, and accordingly, the check controller cannot match the to-be-confirmed data corresponding to the to-be-processed data in the check queue, but can match the data record information corresponding to the to-be-processed data in the history queue. Therefore, the step E1 can be specifically implemented as follows: when the message indicating that the to-be-processed data is processed is received, the check controller first matches the to-be-processed data in the check queue; if the matching is successful, the third current time is taken as the second time stamp of the to-be-processed data that is processed, and the to-be-processed data that is processed, the second time stamp, the first time stamp and the preset timeout threshold are taken as the data record information and sent to the history queue of the work group; if the matching fails, the history queue is matched, and in the case that the data record information corresponding to the to-be-processed data is matched in the history queue, the third current time is taken as the second time stamp of the to-be-processed data that is processed to update the data record information in the history queue.

[0134] Step F2, by the check controller, if it is determined that the target to-be-confirmed data exists in the check queue, the preset value is taken as the second time stamp of the to-be-processed data corresponding to the target to-be-confirmed data, and the to-be-processed data corresponding to the target to-be-confirmed data, the second time stamp, the first time stamp and the preset timeout threshold are taken as the data record information and sent to the history queue of the work group.

[0135] The preset value is a value smaller than the first time stamp of the to-be-processed data corresponding to the target to-be-confirmed data.

[0136] Exemplarily, the preset value can be -1.

[0137] In an implementation manner, the running parameter information includes one or more of a statistical quantity index, a statistical proportion index and a statistical average index; the step of determining, by the statistical controller, the running parameter information of each work group based on the data processing information (i.e., the step E2) can be specifically implemented as the following step e1:

[0138] Step e1, by the statistical controller, the running parameter information of each work group is determined based on the data processing information, and the running parameter information of each work group is stored in the running statistical database.

[0139] The statistical quantity index includes one or more of a total number of data processing information in the preset period, and a number of data processing information of different processing result categories in the preset period; the statistical proportion index includes one or more of a probability of processing completion within the preset timeout threshold, a probability of timeout processing completion, and a probability of loss of to-be-processed data in the preset period; and the statistical average index includes one or more of an average processing completion time and an average timeout time of timeout completion in the preset period.

[0140] Specifically, the probability of processing completion within the preset timeout threshold in the preset period is PF=(CF / C)*100%; the probability of timeout processing completion in the preset period is PO=(CO / C)*100%; and the probability of loss of to-be-processed data in the preset period is PL=(CL / C)*100%.

[0141] The average processing completion time in the preset period is ;

[0142] The average timeout time of timeout completion in the preset period is ;

[0143] C is a total number of data processing information in the preset period; CF is a number of data processing information of a first category of processing result (indicating that to-be-processed data is processed and completed within a preset timeout threshold) in the preset period; CO is a number of data processing information of a second category of processing result (indicating that to-be-processed data is processed and completed after timeout) in the preset period; and CL is a number of data processing information of a third category of processing result (indicating that to-be-processed data is lost) in the preset period.

[0144] In an implementation manner, after the to-be-processed data is sent to the data processing service cluster to enable the data processing service cluster to process the to-be-processed data (namely, S104), the check controller can receive a message indicating that the to-be-processed data is processed and completed, so as to determine that the to-be-processed data is processed and completed, by performing the following steps G1 to G3:

[0145] Step G1: receiving, by the data processing service cluster, the to-be-processed data, and processing the to-be-processed data to obtain processed data.

[0146] Step G2: writing, by the data processing service cluster, the processed data into the monitoring database.

[0147] Step G3: in a case where the processed data is successfully written into the monitoring database, sending, by the monitoring database, a message indicating that the to-be-processed data is processed and completed to the check controller, so as to enable the check controller to determine that the to-be-processed data is processed and completed.

[0148] Specifically, the data processing service cluster includes a plurality of data processing nodes. The data processing node includes a processing controller and a thread pool including a plurality of threads. After the processing controller receives the data to be processed (M ID , M), the thread pool acquires an idle thread and assigns a data processing task. The thread monitors the output value M' obtained after the data M is processed, and writes the processed data (M ID , M') into the monitoring database. After the data is successfully stored, the thread is automatically released. Then, the monitoring database sends a message to the check controller indicating that the data to be processed is processed. The check controller receives the message indicating that the data to be processed is processed, and determines that the data to be processed is processed.

[0149] Figure 4 is a flowchart of another data processing method provided by an embodiment of the present application. As shown in Figure 4 , the method comprises:

[0150] Step 402, acquiring monitoring data through a collection controller in a data collection queue cluster, and assigning a pre-set work group to the monitoring data.

[0151] The work group includes one or more of the following: a collection queue node, a data processing node and a data checking node in the data processing service cluster, and a running statistics node. The data checking node includes a checking queue, and the running statistics node includes a history queue. The history queue is used to store data record information corresponding to the to-be-confirmed data removed from the checking queue by the check controller. The data record information includes to-be-processed data, a first timestamp corresponding to the to-be-processed data, a second timestamp, and a pre-set timeout threshold. The second timestamp represents the time when the to-be-processed data is processed. The data checking queue cluster includes the check controller and the check queue.

[0152] Step 404, generating to-be-processed data based on the monitoring data and the work group to which the monitoring data is assigned through the collection controller, and storing the to-be-processed data in the collection queue node corresponding to the work group.

[0153] Step 406, taking out the to-be-processed data from the collection queue according to the time interval determined by the flow limiter of the work group, and recording the first timestamp of taking out the to-be-processed data from the collection queue.

[0154] Step 408, sending the to-be-processed data and the first timestamp corresponding to the to-be-processed data to the check queue corresponding to the work group as to-be-confirmed data through the flow limiter, and sending the to-be-processed data to the data processing node corresponding to the work group, so that the data processing node processes the to-be-processed data.

[0155] Step 410, receiving the to-be-processed data by the data processing node, and processing the to-be-processed data to obtain processed data.

[0156] The data processing node includes a processing controller and a thread pool, and the thread pool includes a plurality of threads. After the processing controller receives the to-be-processed data (including data identification and monitoring data), the processing controller waits for the thread pool to obtain an idle thread and assign a data processing task. The thread obtains an output value after processing the monitoring data, and then obtains the processed data (including data identification and output value).

[0157] Step 412, writing the processed data into the monitoring database by the data processing node.

[0158] The thread writes the processed data (including data identification and output value) into the monitoring database, and the thread is automatically released after the data is successfully written into the database.

[0159] Step 414, in the case that the processed data is successfully written into the monitoring database, sending a message representing that the to-be-processed data is processed to the check controller through the monitoring database, so that the check controller determines that the to-be-processed data is processed.

[0160] Step 416, through the check controller, if the message representing that the to-be-processed data is processed is received, the third current time is taken as a second time stamp of the processed to-be-processed data, and the processed to-be-processed data, the second time stamp, the first time stamp and the preset timeout threshold are taken as data record information and sent to the history queue of the work group; and the to-be-confirmed data corresponding to the processed to-be-processed data is removed from the check queue corresponding to the work group.

[0161] Step 418, through the check controller, based on the first current time, the preset timeout threshold corresponding to the work group and the first time stamp, determining whether there is target to-be-confirmed data in the check queue.

[0162] Step 420, if there is target to-be-confirmed data, through the check controller, taking a preset value as a second time stamp of the to-be-processed data corresponding to the target to-be-confirmed data, and taking the to-be-processed data corresponding to the target to-be-confirmed data, the second time stamp, the first time stamp and the preset timeout threshold as data record information and sending them to the history queue of the work group.

[0163] The preset value is a value smaller than the first time stamp of the to-be-processed data corresponding to the target to-be-confirmed data.

[0164] Step 422, through the check controller, the to-be-processed data corresponding to the target to-be-confirmed data is re-sent to the data collection queue cluster, so that the to-be-processed data that exceeds the timeout is reprocessed, and the target to-be-confirmed data in the check queue is deleted.

[0165] Step 424, by the statistical controller, determining a processing result category corresponding to the data record information in the historical queue based on the second current time, the data record information stored in the historical queue corresponding to the work group, and a preset timeout tolerance value.

[0166] The processing result category includes a first category, a second category, and a third category; the first category represents that the to-be-processed data is processed within the preset timeout threshold; the second category represents that the to-be-processed data is processed beyond the timeout; and the third category represents that the to-be-processed data is lost.

[0167] Step 426, by the statistical controller, storing the data record information and the processing result category as data processing information in a running statistical database, and determining running parameter information of each work group based on the data processing information in the running statistical database.

[0168] Step 428, by the running parameter controller, pushing the running parameter information of each work group stored in the running statistical database to a target module, so that the target module is updated based on the running parameter information of each work group.

[0169] The target module includes one or more of the following: the acquisition controller and the inspection controller; and the running parameter information is related to the historical processing of the to-be-processed data by the work group.

[0170] It should be noted that the process of the data processing method described above with respect to steps 402 to 428 can refer to the specific implementation process of the above-described embodiments, which will not be described here.

[0171] Corresponding to steps 402 to 428, the present embodiment also provides a schematic block diagram of a corresponding data processing system, as shown in Figure 5 for executing Figure 4 the data processing method described above.

[0172] In the embodiment, the first timestamp of obtaining the to-be-processed data is recorded, the to-be-processed data and the first timestamp corresponding to the to-be-processed data are sent to the check queue as to-be-confirmed data, and the to-be-processed data is sent to the data processing service cluster to enable the data processing service cluster to process the to-be-processed data. Then, based on the first current time, the preset timeout threshold and the first timestamp, the check controller determines whether there is target to-be-confirmed data in the check queue. The target to-be-confirmed data is to-be-confirmed data corresponding to to-be-processed data that has timed out. To-be-confirmed data corresponding to to-be-processed data that has been processed is removed from the check queue. Then, if there is target to-be-confirmed data, the to-be-processed data corresponding to the target to-be-confirmed data is reprocessed, and the target to-be-confirmed data is removed from the check queue. Compared with related monitoring data processing technologies that lack persistent storage and copy protection, the present solution realizes a storage mechanism for to-be-confirmed data corresponding to to-be-processed data and a check mechanism for data processing by adding a check queue and a check controller. In the event of an abnormal event that causes to-be-processed data to be incomplete, the to-be-processed data that has not been processed is found and reprocessed, avoiding loss of to-be-processed data and the inability to recover and process it. Thus, the reliability of processing to-be-processed data is improved, and the problem of poor data processing reliability of related monitoring data processing technologies is solved. In addition, a feedback closed-loop mechanism is established to monitor and count the whole process of monitoring data collection, scheduling, processing and warehousing. Based on the real-time state of the cluster, the data collection load balancing, data processing consumption flow limiting and processing and warehousing completion check operation parameters are dynamically controlled, thereby improving the reliability and controllability of the monitoring data collection and processing process.

[0173] Corresponding to the data processing method provided in the above embodiment, based on the same technical concept, the present embodiment also provides a data processing system, as shown in Figure 6 The data processing system includes a data collection queue cluster 610, a data processing service cluster 620, a check queue 630 and a check controller 640. The data collection queue cluster 610 is configured to obtain to-be-processed data and record a first timestamp of obtaining the to-be-processed data, send the to-be-processed data and the first timestamp corresponding to the to-be-processed data to the check queue as to-be-confirmed data, and send the to-be-processed data to the data processing service cluster.

[0174] The data collection queue cluster 610 is configured to obtain to-be-processed data and record a first timestamp of obtaining the to-be-processed data, send the to-be-processed data and the first timestamp corresponding to the to-be-processed data to the check queue as to-be-confirmed data, and send the to-be-processed data to the data processing service cluster.

[0175] The data processing service cluster 620 is configured to process the to-be-processed data and reprocess to-be-processed data corresponding to target to-be-confirmed data. The target to-be-confirmed data is to-be-confirmed data corresponding to to-be-processed data that has timed out.

[0176] An inspection queue 630 is configured to store to-be-confirmed data;

[0177] An inspection controller 640 is configured to move the to-be-confirmed data corresponding to the processed to-be-processed data and the target to-be-confirmed data out of the inspection queue, and determine whether the target to-be-confirmed data exists in the inspection queue based on the first current time, the preset timeout threshold, and the first timestamp.

[0178] Specifically, the target to-be-confirmed data can be sent to the data collection queue cluster 610 by the inspection controller 640, so that the data collection queue cluster 610 reprocesses the to-be-processed data corresponding to the target to-be-confirmed data.

[0179] In an implementation manner, the data collection queue cluster 610 includes a collection controller;

[0180] The collection controller is configured to acquire monitoring data, and assign the monitoring data to a preset workgroup; generate to-be-processed data based on the monitoring data and the workgroup to which the monitoring data is assigned, and store the to-be-processed data in the data collection queue cluster;

[0181] The workgroup includes a data processing node in the data processing service cluster.

[0182] The data collection queue cluster 610 is specifically configured to send the to-be-processed data to the data processing node corresponding to the workgroup.

[0183] The data processing node is configured to process the to-be-processed data.

[0184] In an implementation manner, the data processing system further includes a running parameter controller.

[0185] The running parameter controller is configured to push the running parameter information of each workgroup pre-stored in the running statistics database to a target module, so that the target module performs corresponding update based on the running parameter information of each workgroup; the running parameter information is related to a historical processing situation of the workgroup to the to-be-processed data.

[0186] The target module includes one or more of the following: the collection controller and the inspection controller.

[0187] In an implementation manner, the workgroup further includes a data inspection node, and the data inspection node includes an inspection queue; the inspection controller 640 is specifically configured to:

[0188] Move the to-be-confirmed data corresponding to the processed to-be-processed data out of the inspection queue corresponding to the workgroup, and determine whether the to-be-processed data corresponding to the to-be-confirmed data in the inspection queue is timed out based on the first current time, the preset timeout threshold corresponding to the workgroup, and the first timestamp of the to-be-confirmed data in the inspection queue.

[0189] The inspection controller 640 is further configured to:

[0190] update the preset timeout threshold of each workgroup based on the operation parameter information of each workgroup and the preset timeout rate.

[0191] In an implementation manner, the collection controller is specifically configured to: based on the preset weight of each workgroup, the preset total weight, and the preset operation weight of each workgroup, determine the workgroup to which the monitoring data is allocated by using a load balancing algorithm.

[0192] The collection controller is further configured to: update the preset weight and the preset total weight of each workgroup based on the operation parameter information of each workgroup.

[0193] In an implementation manner, the data collection queue cluster 610 further includes a collection queue node; the workgroup further includes the collection queue node; and the collection queue node includes a flow limiter and a collection queue.

[0194] The collection queue is configured to store the to-be-processed data.

[0195] The flow limiter is configured to: according to a time interval determined by the flow limiter, take the to-be-processed data from the collection queue and record a first time stamp of taking the to-be-processed data from the collection queue; send the to-be-processed data and the first time stamp corresponding to the to-be-processed data to the inspection queue corresponding to the workgroup as the to-be-confirmed data, and send the to-be-processed data to the data processing node corresponding to the workgroup, so that the data processing node processes the to-be-processed data.

[0196] In an implementation manner, the flow limiter is specifically configured to: based on a preset flow limiting frequency, determine the time interval, and according to the time interval, take the to-be-processed data from the collection queue.

[0197] The flow limiter is further configured to: update the preset flow limiting frequency of each workgroup based on the operation parameter information of each workgroup.

[0198] In an implementation manner, the data processing system further includes an operation statistics node and a statistics controller, and the operation statistics node includes a history queue.

[0199] The history queue is configured to store data record information corresponding to the to-be-confirmed data removed from the inspection queue corresponding to the workgroup by the inspection controller.

[0200] The data record information includes the to-be-processed data, the first time stamp corresponding to the to-be-processed data, a second time stamp, and a preset timeout threshold; and the second time stamp represents a time when the to-be-processed data is processed.

[0201] The statistical controller is configured to determine a processing result category corresponding to the data record information in the historical queue based on the second current time, the data record information stored in the historical queue corresponding to the work group, and a preset timeout tolerance value; store the data record information and the processing result category as data processing information in the operation statistical database; and determine operation parameter information of each work group based on the data processing information in the operation statistical database.

[0202] The processing result category includes a first category, a second category, and a third category; the first category indicates that the to-be-processed data is processed within the preset timeout threshold; the second category indicates that the to-be-processed data is processed beyond the timeout; and the third category indicates that the to-be-processed data is lost.

[0203] In an implementation manner, the checking controller 640 is further configured to: if the message indicating that the to-be-processed data is processed is received, take the third current time as a second time stamp of the to-be-processed data that is processed, and send the to-be-processed data that is processed, the second time stamp, the first time stamp, and the preset timeout threshold as data record information to the historical queue of the work group; if it is determined that the target to-be-confirmed data exists in the checking queue, take a preset value as a second time stamp of the to-be-processed data corresponding to the target to-be-confirmed data, and send the to-be-processed data corresponding to the target to-be-confirmed data, the second time stamp, the first time stamp, and the preset timeout threshold as data record information to the historical queue of the work group; and the preset value is a value smaller than the first time stamp of the to-be-processed data corresponding to the target to-be-confirmed data.

[0204] In an implementation manner, the operation parameter information includes one or more of a statistical quantity index, a statistical proportion index, and a statistical average index; and the statistical controller is specifically configured to:

[0205] determine the operation parameter information of each work group based on the data processing information, and store the operation parameter information of each work group in the operation statistical database;

[0206] The statistical quantity index includes one or more of a total number of the data processing information, a number of the data processing information of different processing result categories in a preset period; the statistical proportion index includes one or more of a probability of processing completion within the preset timeout threshold, a probability of processing completion beyond the timeout, and a probability of loss of the to-be-processed data in the preset period; and the statistical average index includes one or more of an average processing completion time and an average timeout time of timeout completion in the preset period.

[0207] In an implementation manner, the data processing system further includes a monitoring database.

[0208] The data processing service cluster 620 is specifically configured to: receive the to-be-processed data, process the to-be-processed data to obtain processed data, and write the processed data into the monitoring database;

[0209] A monitoring database is configured to store the processed data, and send a message to the check controller indicating that the to-be-processed data is processed successfully, so that the check controller determines that the to-be-processed data is processed.

[0210] In this embodiment, the to-be-processed data is first acquired, and a first timestamp of acquiring the to-be-processed data is recorded; the to-be-processed data and the first timestamp corresponding to the to-be-processed data are sent to the check queue as to-be-confirmed data, and the to-be-processed data is sent to the data processing service cluster to process the to-be-processed data; then, the check controller determines whether there is target to-be-confirmed data in the check queue based on the first current time, the preset timeout threshold and the first timestamp; the target to-be-confirmed data is the to-be-confirmed data corresponding to the to-be-processed data that exceeds the timeout; the to-be-confirmed data corresponding to the to-be-processed data that is processed is removed from the check queue; then, if there is target to-be-confirmed data, the to-be-processed data corresponding to the target to-be-confirmed data is reprocessed, and the target to-be-confirmed data is removed from the check queue. Compared with related monitoring data processing technology, there is no persistent storage and copy protection. This scheme realizes the storage mechanism of the to-be-confirmed data corresponding to the to-be-processed data and the check mechanism of the data processing situation by adding the check queue and the check controller, can find the to-be-processed data that is not processed when an abnormal event occurs, reprocess it, avoid the loss of to-be-processed data and the inability to recover and process, thereby improving the reliability of processing the to-be-processed data, and solving the problem of poor data processing reliability of related monitoring data processing technology.

[0211] According to the same technical concept, the data processing method provided in the above embodiment, the present embodiment also provides a data processing device, Figure 7 is a structural schematic diagram of the data processing device according to the embodiment of the present application, which is used to execute Figures 1 to 4 The data processing method described above, as shown in Figure 7 The data processing device includes an acquisition module 710, a sending module 720, a determination module 730 and a reprocessing module 740.

[0212] The acquisition module 710 is configured to acquire to-be-processed data and record a first timestamp of acquiring the to-be-processed data;

[0213] The sending module 720 is configured to send the to-be-processed data and the first timestamp corresponding to the to-be-processed data to the check queue as to-be-confirmed data, and send the to-be-processed data to the data processing service cluster to process the to-be-processed data;

[0214] The determination module 730 is configured to determine whether there is target to-be-confirmed data in the check queue based on the current time, the preset timeout threshold, and the first timestamp; the target to-be-confirmed data is to-be-confirmed data corresponding to the to-be-processed data that is timed out; and the to-be-confirmed data corresponding to the to-be-processed data that is processed completely is removed from the check queue.

[0215] The reprocessing module 740 is configured to reprocess the to-be-processed data corresponding to the target to-be-confirmed data if the target to-be-confirmed data exists, and remove the target to-be-confirmed data from the check queue.

[0216] In an implementation manner, the data collection queue cluster includes a collection controller; and the acquisition module 710 includes:

[0217] The allocation unit is configured to acquire the monitoring data by the collection controller in the data collection queue cluster, and allocate the monitoring data to a pre-set work group.

[0218] The work group includes one or more of the following: a collection queue node, a data processing node in the data processing service cluster, and a data check node; and the data check node includes a check queue.

[0219] The storage unit is configured to generate to-be-processed data based on the monitoring data and the work group to which the monitoring data is allocated by the collection controller, and store the to-be-processed data in the data collection queue cluster.

[0220] The recording unit is configured to acquire the to-be-processed data stored in the data collection queue cluster by the data collection queue cluster.

[0221] The sending module 720 is specifically configured to send the to-be-processed data to the data processing node corresponding to the work group, so that the data processing node corresponding to the work group processes the to-be-processed data.

[0222] In an implementation manner, the data processing apparatus further includes a pushing module.

[0223] The pushing module is configured to push the running parameter information of each work group pre-stored in the running statistical database to a target module by the running parameter controller, so that the target module is updated based on the running parameter information of each work group; and the running parameter information is related to the historical processing situation of the work group to the to-be-processed data.

[0224] The target module includes one or more of the following: the collection controller and the check controller.

[0225] In an implementation manner, the working group further includes a data checking node, and the data checking node includes a checking queue; the determination module 730 is specifically configured to determine, by the checking controller, whether the target to-be-confirmed data exists in the checking queue based on the first current time, the preset timeout threshold corresponding to the working group, and the first timestamp.

[0226] The pushing module is specifically configured to update, by the checking controller, the preset timeout threshold of each working group based on the running parameter information of each working group and the preset timeout rate.

[0227] In an implementation manner, the allocation unit is specifically configured to:

[0228] determine, by the collection controller, the working group to which the monitoring data is allocated by using a load balancing algorithm based on the preset weight of each working group, the preset total weight, and the preset running weight of each working group;

[0229] The pushing module is specifically configured to update, by the collection controller, the preset weight and the preset total weight of each working group based on the running parameter information of each working group.

[0230] In an implementation manner, the data collection queue cluster further includes a collection queue node; the working group further includes the collection queue node; the collection queue node includes a flow limiter and a collection queue; and the recording unit includes a recording subunit. The storage unit is specifically configured to:

[0231] store the to-be-processed data into the collection queue corresponding to the working group;

[0232] The recording subunit is configured to:

[0233] take out, by the flow limiter of the working group, the to-be-processed data from the collection queue corresponding to the working group according to the time interval determined by the flow limiter, and record a first timestamp of taking out the to-be-processed data from the collection queue;

[0234] The sending module 720 is specifically configured to:

[0235] take out, by the flow limiter of the working group, the to-be-processed data from the collection queue corresponding to the working group according to the time interval determined by the flow limiter, and record a first timestamp of taking out the to-be-processed data from the collection queue;

[0236] In an implementation manner, the recording subunit is specifically configured to:

[0237] determine, by the flow limiter, the time interval based on a preset flow limiting frequency, and take out the to-be-processed data from the collection queue according to the time interval;

[0238] The pushing module is specifically configured to update the preset flow limiting frequency of each work group based on the running parameter information of each work group through the flow limiter.

[0239] In an implementation manner, the work group further includes a running statistics node, and the running statistics node includes a history queue; the history queue is configured to store data record information corresponding to the to-be-confirmed data removed from the inspection queue corresponding to the work group by the inspection controller; the data record information includes to-be-processed data, a first timestamp corresponding to the to-be-processed data, a second timestamp, and a preset timeout threshold; the second timestamp represents a time when the to-be-processed data is processed; and the data processing apparatus further includes a statistics module.

[0240] The determining category unit is configured to determine a processing result category corresponding to the data record information in the history queue by the statistics controller based on the second current time, the data record information stored in the history queue corresponding to the work group, and a preset timeout tolerance value.

[0241] The processing result category includes a first category, a second category, and a third category; the first category represents that the to-be-processed data is processed within the preset timeout threshold; the second category represents that the to-be-processed data is processed beyond the timeout; and the third category represents that the to-be-processed data is lost.

[0242] The determining information unit is configured to store the data record information and the processing result category as data processing information in the running statistics database by the statistics controller, and determine the running parameter information of each work group based on the data processing information in the running statistics database.

[0243] In an implementation manner, the data processing apparatus further includes a determining information module. The determining information module is configured to:

[0244] If the message representing that the to-be-processed data is processed is received by the inspection controller, the third current time is taken as the second timestamp of the to-be-processed data processed, and the to-be-processed data processed, the second timestamp, the first timestamp, and the preset timeout threshold are taken as the data record information and sent to the history queue of the work group.

[0245] If it is determined that the target to-be-confirmed data exists in the inspection queue by the inspection controller, the preset value is taken as the second timestamp of the to-be-processed data corresponding to the target to-be-confirmed data, and the to-be-processed data corresponding to the target to-be-confirmed data, the second timestamp, the first timestamp, and the preset timeout threshold are taken as the data record information and sent to the history queue of the work group; and the to-be-confirmed data corresponding to the to-be-processed data in the inspection queue is deleted; the preset value is a value smaller than the first timestamp of the to-be-processed data corresponding to the target to-be-confirmed data.

[0246] In an implementation manner, the running parameter information comprises one or more of a statistical quantity index, a statistical proportion index and a statistical average index; the determination information unit is specifically configured to:

[0247] The statistical controller is configured to determine the running parameter information of each workgroup based on the data processing information, and store the running parameter information of each workgroup into the running statistics database.

[0248] The statistical quantity index comprises one or more of a total number of the data processing information, a number of the data processing information of different processing result categories in a preset period; the statistical proportion index comprises one or more of a probability of being processed within a preset timeout threshold, a probability of being processed overtime and a probability of being lost in the preset period; and the statistical average index comprises one or more of an average processing completion time and an average overtime time of overtime completion in the preset period.

[0249] In an implementation manner, the data processing apparatus further comprises a message sending module. The message sending module is configured to:

[0250] The data processing service cluster is configured to receive the to-be-processed data and process the to-be-processed data to obtain processed data.

[0251] The data processing service cluster is configured to write the processed data into the monitoring database.

[0252] In a case that the processed data is successfully written into the monitoring database, the monitoring database is configured to send a message indicating that the to-be-processed data is processed to the check controller, so as to make the check controller determine that the to-be-processed data is processed.

[0253] In this embodiment, the first timestamp of obtaining the to-be-processed data is recorded first, and the to-be-processed data and the first timestamp corresponding to the to-be-processed data are sent to the check queue as to-be-confirmed data, and the to-be-processed data is sent to the data processing service cluster to make the data processing service cluster process the to-be-processed data; then, based on the first current time, the preset timeout threshold and the first timestamp, the check controller determines whether there is target to-be-confirmed data in the check queue; wherein, the target to-be-confirmed data is the to-be-confirmed data corresponding to the to-be-processed data whose processing is timed out; the to-be-confirmed data corresponding to the to-be-processed data that has been processed is removed from the check queue; then, if there is target to-be-confirmed data, the to-be-processed data corresponding to the target to-be-confirmed data is reprocessed, and the target to-be-confirmed data is removed from the check queue. Compared with related monitoring data processing technologies, there is a lack of persistent storage and copy protection. In this scheme, the check queue and the check controller are added to realize the storage mechanism of the to-be-confirmed data corresponding to the to-be-processed data and the checking mechanism of the data processing situation. When an abnormal event occurs to cause the to-be-processed data to be not processed completely, the to-be-processed data that is not processed completely can be found and reprocessed, avoiding the loss of to-be-processed data and the inability to recover and process, thereby improving the reliability of processing the to-be-processed data and solving the problem of poor data processing reliability of related monitoring data processing technologies.

[0254] Those skilled in the art should understand that the above data processing device can be used to implement the data processing method described above, and the detailed description should be similar to the method part described above. To avoid repetition, it will not be described here.

[0255] Based on the same technical concept, the embodiments of the present application also provide an electronic device for executing the above data processing method, Figure 8 To implement an electronic device of various embodiments of the present application. The electronic device can have a big difference due to different configurations or performances, and can include a processor 810, a communications interface 820, a memory 830 and a communications bus 840. The processor 810, the communications interface 820 and the memory 830 complete mutual communication through the communications bus 840. The processor 810 can call the computer program stored on the memory 830 and executable on the processor 810 to execute the following steps:

[0256] Obtaining to-be-processed data and recording the first timestamp of obtaining the to-be-processed data;

[0257] send the to-be-processed data and the first timestamp corresponding to the to-be-processed data to the check queue as to-be-confirmed data, and send the to-be-processed data to the data processing service cluster to enable the data processing service cluster to process the to-be-processed data;

[0258] determine, by the check controller, whether there is target to-be-confirmed data in the check queue based on the first current time, the preset timeout threshold, and the first timestamp, wherein the target to-be-confirmed data is to-be-confirmed data corresponding to to-be-processed data that has timed out, and the to-be-confirmed data corresponding to the to-be-processed data that has been processed is removed from the check queue;

[0259] If the target to-be-confirmed data exists, reprocess the to-be-processed data corresponding to the target to-be-confirmed data, and remove the target to-be-confirmed data from the check queue.

[0260] In this embodiment, the to-be-processed data is first acquired, and the first timestamp at which the to-be-processed data is acquired is recorded. The to-be-processed data and the first timestamp corresponding to the to-be-processed data are sent to the check queue as to-be-confirmed data, and the to-be-processed data is sent to the data processing service cluster to enable the data processing service cluster to process the to-be-processed data. Then, the check controller determines whether there is target to-be-confirmed data in the check queue based on the first current time, the preset timeout threshold, and the first timestamp. The target to-be-confirmed data is to-be-confirmed data corresponding to to-be-processed data that has timed out, and the to-be-confirmed data corresponding to the to-be-processed data that has been processed is removed from the check queue. If the target to-be-confirmed data exists, the to-be-processed data corresponding to the target to-be-confirmed data is reprocessed, and the target to-be-confirmed data is removed from the check queue. Compared with related monitoring data processing technologies that lack persistent storage and replica protection, this solution adds a check queue and a check controller to implement a storage mechanism for to-be-confirmed data corresponding to to-be-processed data and a check mechanism for data processing, so that when an abnormal event occurs to cause to-be-processed data that has not been processed to exist, the to-be-processed data that has not been processed is found and reprocessed, thereby avoiding loss of the to-be-processed data and enabling recovery and processing of the to-be-processed data. Therefore, the reliability of processing to-be-processed data is improved, and the problem of poor reliability of data processing in related monitoring data processing technologies is solved.

[0261] The specific execution steps can refer to the steps of the data processing method embodiments described above and achieve the same technical effects. To avoid repetition, they will not be described here again.

[0262] It should be noted that the electronic device in the embodiments of the present application includes a server, a terminal, or other devices other than a terminal.

[0263] The above electronic device structure does not constitute a limitation on the electronic device, which can include more or fewer components than those shown, or combine some components, or have different arrangements of components, for example, the input unit can include a Graphics Processing Unit (GPU) and a microphone, and the display unit can be configured with a display panel in the form of a liquid crystal display, an organic light-emitting diode, etc. The user input unit includes at least one of a touch panel and other input devices. The touch panel is also referred to as a touch screen. Other input devices can include, but are not limited to, a physical keyboard, function keys (such as volume control keys, on-off keys, etc.), trackballs, mice, joysticks, and the like, which are not described here.

[0264] The memory can be used to store software programs and various data. The memory can mainly include a first storage area storing programs or instructions and a second storage area storing data, wherein the first storage area can store an operating system, application programs or instructions required by at least one function (such as a sound playing function, an image playing function, etc.), and the like. In addition, the memory can include a volatile memory or a non-volatile memory, or the memory can include both volatile and non-volatile memories. Among them, the non-volatile memory can be a Read-Only Memory (ROM), a Programmable ROM (PROM), an Erasable PROM (EPROM), an Electrically EPROM (EEPROM), or a flash memory. The volatile memory can be a Random Access Memory (RAM), a Static RAM (SRAM), a Dynamic RAM (DRAM), a Synchronous DRAM (SDRAM), a Double Data Rate SDRAM (DDR SDRAM), an Enhanced SDRAM (ESDRAM), a Synchlink DRAM (SLDRAM), and a Direct Rambus RAM (DRRAM).

[0265] The processor can include one or more processing units; optionally, the processor integrates an application processor and a modem processor, wherein the application processor mainly processes operations related to an operating system, a user interface, and an application program, and the modem processor mainly processes a wireless communication signal, such as a baseband processor. It can be understood that the above-mentioned modem processor can also not be integrated into the processor.

[0266] The embodiment of the present application further provides a storage medium, and the storage medium stores computer executable instructions. The computer executable instructions are executed by a processor to implement various processes of the above-mentioned data processing method embodiment, and achieve the same technical effects. To avoid repetition, details are not described herein.

[0267] The processor is a processor in the electronic device in the above-mentioned embodiment. The storage medium includes a computer readable storage medium, such as a computer read-only memory (Read-Only Memory, ROM), a random access memory (Random Access Memory, RAM), a magnetic disk or an optical disk, and the like.

[0268] The embodiment of the present application further provides a chip, and the chip includes a processor and a communication interface. The communication interface is coupled with the processor. The processor is configured to execute a program or an instruction to implement various processes of the above-mentioned permission management method embodiment, and achieve the same technical effects. To avoid repetition, details are not described herein.

[0269] It should be understood that the chip mentioned in the embodiment of the present application can also be referred to as a system level chip, a system chip, a chip system, or a system on chip, and the like.

[0270] The embodiment of the present application further provides a computer program product, and the computer program product includes a computer program. The computer program is executed by a processor to implement various processes of the above-mentioned data processing method embodiment, and achieve the same technical effects. To avoid repetition, details are not described herein.

[0271] It should be noted that, in the present document, the terms "comprises", "comprising", or any other variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but can also include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element preceded by "comprises... a" does not, without more constraints, foreclose the existence of additional identical elements in the process, method, article, or apparatus that comprises the element. Also, it should be noted that the scope of the methods and apparatus of the present embodiments are not limited by the order of the steps or processes recited, and that a recited step or process can occur before, after, concurrently, in random order, or in reverse order to other steps or processes unless explicitly stated otherwise. Furthermore, features described with respect to certain examples can be combined in other examples.

[0272] From the above description of the embodiments, it is clear that the above-described method of the embodiments can be realized by means of software and the necessary universal hardware platform, of course, it can also be realized by hardware, but in many cases, the former is a better embodiment. Based on such understanding, the technical solutions of the present application can be embodied in the form of a software product in essence or in the form of a part of the prior art. The computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a plurality of instructions for causing a terminal (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the method of each embodiment of the present application.

[0273] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-described specific embodiments, and the above-described specific embodiments are merely illustrative, rather than limiting. Those skilled in the art can make many forms under the inspiration of the present application without departing from the scope of the present application and the scope of protection of the claims.

Claims

1. A data processing method, characterized in that, The method includes: The monitoring data is acquired through the acquisition controller in the data acquisition queue cluster, and a pre-set work group is assigned to the monitoring data; wherein, the work group includes data processing nodes in the data processing service cluster; The acquisition controller generates data to be processed based on the monitoring data and the workgroup to which the monitoring data is assigned, and stores the data to be processed into the data acquisition queue cluster. The system obtains the data to be processed stored in the data acquisition queue cluster and records the first timestamp of obtaining the data to be processed. The data to be processed and the first timestamp corresponding to the data to be processed are sent as data to be confirmed to the inspection queue, and the data to be processed is sent to the data processing node corresponding to the work group so that the data processing node corresponding to the work group can process the data to be processed. By checking the controller, based on the first current time, the preset timeout threshold, and the first timestamp, it is determined whether there is target data to be confirmed in the check queue; wherein, the target data to be confirmed is the data to be confirmed corresponding to the data to be processed that has timed out; the data to be confirmed corresponding to the data to be processed that has been completed is removed from the check queue; If the target data to be confirmed exists, then the data to be processed corresponding to the target data to be confirmed is reprocessed, and the target data to be confirmed is removed from the inspection queue.

2. The method according to claim 1, characterized in that, The method further includes: The runtime parameter controller pushes the runtime parameter information of each working group, which is pre-stored in the runtime statistics database, to the target module, so that the target module can update accordingly based on the runtime parameter information of each working group; the runtime parameter information is related to the historical processing status of the working group on the data to be processed. The target module includes one or more of the following: the acquisition controller and the inspection controller.

3. The method according to claim 2, characterized in that, The working group also includes a data inspection node, which includes the inspection queue; The step of determining whether target data to be confirmed exists in the inspection queue by checking the controller, based on the first current time, a preset timeout threshold, and the first timestamp, includes: The step involves checking the controller to determine whether the target data to be confirmed exists in the check queue based on the first current time, the preset timeout threshold corresponding to the work group, and the first timestamp. If the target module includes the inspection controller, the target module performs corresponding updates based on the operating parameter information of each of the working groups, including: The inspection controller updates the preset timeout threshold for each workgroup based on the operating parameter information and preset timeout rate of each workgroup.

4. The method according to claim 2, characterized in that, The process of assigning a pre-set workgroup to the monitoring data includes: The acquisition controller uses a load balancing algorithm to determine the workgroups assigned to the monitoring data, based on the preset weights, preset total weights, and preset operating weights of each workgroup. If the target module includes the acquisition controller, the target module performs corresponding updates based on the operating parameter information of each of the working groups, including: The acquisition controller updates the preset weight and the preset total weight of each working group based on the operating parameter information of each working group.

5. The method according to claim 2, characterized in that, The data acquisition queue cluster further includes acquisition queue nodes; the workgroup further includes the acquisition queue nodes; the acquisition queue nodes include a rate limiter and an acquisition queue; storing the data to be processed into the data acquisition queue cluster includes: The data to be processed is stored in the collection queue corresponding to the work group; The step of acquiring the data to be processed stored in the data acquisition queue cluster through the data acquisition queue cluster and recording the first timestamp of acquiring the data to be processed includes: The data to be processed is retrieved from the acquisition queue corresponding to the work group according to the time interval determined by the current limiter of the work group, and the first timestamp of retrieving the data to be processed from the acquisition queue is recorded. Accordingly, sending the data to be processed and the first timestamp corresponding to the data to be processed as confirmation data to the inspection queue, and sending the data to be processed to the data processing node corresponding to the workgroup, so that the data processing node corresponding to the workgroup processes the data to be processed, includes: The data to be processed and the first timestamp corresponding to the data to be processed are sent as the data to be confirmed to the check queue corresponding to the work group through the rate limiter corresponding to the work group, and the data to be processed is sent to the data processing node corresponding to the work group so that the data processing node can process the data to be processed.

6. The method according to claim 5, characterized in that, The step of retrieving the data to be processed from the acquisition queue according to the time interval determined by the current limiter includes: The time interval is determined by the current limiter based on a preset current limiting frequency, and the data to be processed is retrieved from the acquisition queue according to the time interval. The target module also includes the current limiter, which updates accordingly based on the operating parameter information of each of the working groups, including: The preset current limiting frequency of each working group is updated using the current limiter based on the operating parameter information of each working group.

7. The method according to claim 1, characterized in that, The working group further includes a running statistics node, which includes a historical queue. The historical queue is used to store data record information corresponding to the data to be confirmed that has been removed from the inspection queue corresponding to the working group by the inspection controller. The data record information includes the data to be processed, the first timestamp, the second timestamp corresponding to the data to be processed, and the preset timeout threshold. The second timestamp represents the time when the data to be processed was completed; The method further includes: The statistical controller determines the processing result category corresponding to the data record information in the historical queue based on the second current time, the data record information stored in the historical queue corresponding to the work group, and the preset timeout tolerance value. The processing result categories include a first category, a second category, and a third category; the first category indicates that the data to be processed has been processed within the preset timeout threshold; the second category indicates that the data to be processed has been processed after a timeout; and the third category indicates that the data to be processed has been lost. The statistical controller stores the data recording information and the processing result category as data processing information in the operational statistics database, and determines the operational parameter information of each working group based on the data processing information in the operational statistics database.

8. The method according to claim 7, characterized in that, Before determining the processing result category corresponding to the data record information in the historical queue through the statistical controller, based on the second current time, the data record information stored in the historical queue corresponding to the workgroup, and a preset timeout tolerance value, the method further includes: If the controller receives a message indicating that the data to be processed has been completed, it will use the third current time as the second timestamp of the data to be processed that has been completed, and send the data to be processed that has been completed, the second timestamp, the first timestamp, and the preset timeout threshold as the data record information into the historical queue of the working group. If the controller determines that the target data to be confirmed exists in the inspection queue, a preset value is used as the second timestamp of the data to be processed corresponding to the target data to be confirmed, and the data to be processed corresponding to the target data to be confirmed, the second timestamp, the first timestamp, and the preset timeout threshold are sent as the data record information to the historical queue of the work group; the preset value is less than the value of the first timestamp of the data to be processed corresponding to the target data to be confirmed.

9. The method according to claim 7, characterized in that, The operational parameter information includes one or more of statistical quantity indicators, statistical proportion indicators, and statistical average indicators; the determination of the operational parameter information for each working group based on the data processing information in the operational statistics database includes: The statistical controller determines the operating parameter information of each working group based on the data processing information and stores the operating parameter information of each working group in the operating statistics database. The statistical quantity indicators include one or more of the total number of data processing information items within a preset period and the number of data processing information items of different processing result categories; the statistical proportion indicators include one or more of the probability of processing completed within the preset timeout threshold, the probability of processing completed after timeout, and the probability of data loss during processing within the preset period; the statistical average indicators include one or more of the average processing completion time and the average timeout time for processing completed after timeout within the preset period.

10. The method according to claim 1, characterized in that, After sending the data to be processed to the data processing node corresponding to the workgroup, so that the data processing node corresponding to the workgroup can process the data to be processed, the method further includes: The data processing service cluster receives the data to be processed and processes it to obtain the processed data. The processed data is written into the monitoring database through the data processing service cluster. If the processed data is successfully written to the monitoring database, a message indicating that the data to be processed has been processed is sent to the inspection controller through the monitoring database, so that the inspection controller can determine that the data to be processed has been processed.

11. A data processing system, characterized in that, The system includes: A data acquisition queue cluster is used to acquire monitoring data through an acquisition controller within the cluster and to assign pre-set workgroups to the monitoring data. The workgroups include data processing nodes within a data processing service cluster. The acquisition controller generates data to be processed based on the monitoring data and the workgroups to which the monitoring data is assigned, and stores the data to be processed in the data acquisition queue cluster. The data acquisition queue cluster acquires the data to be processed stored therein and records a first timestamp of the acquired data. The data to be processed and the corresponding first timestamp are sent as confirmation data to an inspection queue, and the data to be processed is sent to the data processing node corresponding to the workgroup. The data processing service cluster is used to process the data to be processed through the data processing nodes corresponding to the workgroup, and to reprocess the data to be processed corresponding to the target data to be confirmed; wherein, the target data to be confirmed is the data to be confirmed corresponding to the data to be processed that has timed out. A check queue is used to store the data to be confirmed; The check controller is configured to remove the pending confirmation data and the target pending confirmation data corresponding to the processed pending data from the check queue, and to determine whether the target pending confirmation data exists in the check queue based on the first current time, the preset timeout threshold and the first timestamp.

12. The system according to claim 11, characterized in that, The system also includes: The operation parameter controller is used to push the operation parameter information of each workgroup, which is pre-stored in the operation statistics database, to the target module, so that the target module can make corresponding updates based on the operation parameter information of each workgroup; the operation parameter information is related to the historical processing status of the workgroup on the data to be processed; The target module includes one or more of the following: the acquisition controller and the inspection controller.

13. A data processing apparatus, characterized in that, The device includes: The acquisition module is used to acquire monitoring data through the acquisition controller in the data acquisition queue cluster, and assign a pre-set workgroup to the monitoring data; wherein, the workgroup includes data processing nodes in the data processing service cluster; through the acquisition controller, based on the monitoring data and the workgroup to which the monitoring data is assigned, generate data to be processed, and store the data to be processed in the data acquisition queue cluster; through the data acquisition queue cluster, acquire the data to be processed stored in the data acquisition queue cluster, and record the first timestamp of acquiring the data to be processed; The sending module is configured to send the data to be processed and the first timestamp corresponding to the data to be processed as data to be confirmed to the inspection queue, and to send the data to be processed to the data processing node corresponding to the work group, so that the data processing node corresponding to the work group can process the data to be processed. The determination module is used to determine, by checking the controller, whether there is target data to be confirmed in the inspection queue based on a first current time, a preset timeout threshold and a first timestamp; wherein, the target data to be confirmed is the data to be confirmed corresponding to the data to be processed that has timed out; the data to be confirmed corresponding to the data to be processed that has been processed is removed from the inspection queue; The reprocessing module is used to reprocess the data to be processed corresponding to the target data to be confirmed if the target data to be confirmed exists, and to remove the target data to be confirmed from the inspection queue.

14. An electronic device, characterized in that, include: processor; as well as A memory configured to store computer-executable instructions configured to be executed by the processor, the executable instructions including instructions for performing a data processing method as described in any one of claims 1-10.

15. A storage medium, characterized in that, The storage medium is used to store computer-executable instructions that cause a computer to perform the data processing method as described in any one of claims 1-10.

16. A computer program product, characterized in that, It includes a computer program that, when executed by a processor, implements the data processing method as described in any one of claims 1-10.

Images